6 ms·

New Statistical De-minifier and De-obfuscator for JavaScript

173 points by mvechev 12y ago

infogulch 12y ago
Whoa, it even infers local variable names automatically.
joliss 12y ago
About: http://www.srl.inf.ethz.ch/jsnice.php http://www.srl.inf.ethz.ch/jsnice.php Apparently this is a machine learning research project. It seems that this would be quite useful for looking at the code of closed-source webapps. Would love to see it open-sourced or available as a service (via an API perhaps). Think auto-deminifying browser extension.
- geetee 12y ago
  On average, more than 60% of the identifiers are recovered to the same name as before the minification process. I wonder how much of that 60% comes from common libraries like jQuery or Underscore. Still, a neat project.
  - veselin 12y ago
    We tried to evaluate on data that is as much as possible independent from the training data. So we evaluate on projects outside of github.
leeoniya 12y ago
didn't do too well on http://js1k.com/2014-dragons/details/1903 http://js1k.com/2014-dragons/details/1903 :) but processed jQuery relatively fast.
veselin 12y ago
Yes, it is statistical. A browser extension would be cool!
yconst 12y ago
Very interesting project. Would be great to get some more info on the actual algorithms being used. The about page offers relatively limited info.
mvechev 12y ago
Thank you all for the comments...keep them coming. we will definitely soon provide more details on how the overall system works...
pbielik 12y ago
Handy tool to have. Works suprisingly good.
mallamanis 12y ago
Naturalize ( http://groups.inf.ed.ac.uk/naturalize/ http://groups.inf.ed.ac.uk/naturalize/ ) is another related machine learning based tool that suggests appropriate identifier names for Java. Details on the implementation and an evaluation may be found here http://arxiv.org/abs/1402.4182 http://arxiv.org/abs/1402.4182
pajtai 12y ago
I took the code presented and put it into packer - http://dean.edwards.name/packer/ http://dean.edwards.name/packer/ , and the "nice" output was not very helpful in that it still looked obfuscated. Maybe I'm misunderstanding something. So the input to js nice was the packed generateSeries function: eval(function(p,a,c,k,e,r){e=function(c) {return(c<a?'':e(parseInt(c/a)))+((c=c%a)>35? String.fromCharCode(c+29):c.toString(36))};if(!''.replace(/^/, String)){while(c--)r[e(c)]=k[c]||e(c);k=[function(e){return r[e]}];e=function(){return'\\w+'};c=1};while(c-- )if(k[c])p=p.replace(new RegExp('\\b'+e(c)+'\\b','g'),k[c]);return p}('4 3(e){2 t=[];2 n=f+e;2 r=6+e;k(i=1;i<=6;i++){2 s=5.y(5.a()*(r- n+1)+n);t.b([i,s]);n++;r++}c t}$(d).B(4(){2 e=3(0);2 t=3(g);$.h($("#j"),[{7:"l",8:e},{7:"o",8:t}],{p:{q:{u: ["#v","#w"]}},x:{9:z},A: {9:m}})})',38,38,'||var|generateSeries|function|Math|200|label |data|ticks|random|push|return|document||100|300|plot||flotcon tainer|for|data1|10||data2|grid|backgroundColor||||colors|D1D1 D1|7A7A7A|xaxis|floor|20|yaxis|ready'.split('|'),0,{}))
- mVChr 12y ago
  Noticed that too, was kind of happy about it. For most projects I use uglify or Google Closure Compiler and the like, which it seems to do a good job with. When I really want something obfuscated to make it as difficult as possible for someone to figure out what's going on behind the scenes though I use packer and am glad this tool couldn't figure out what to do with it.
- spicyj 12y ago
  http://jsbeautifier.org/ http://jsbeautifier.org/ has no problem deobfuscating that.
- adrenalinup 12y ago
  It's easy to depack even the packer's "Base 64 encode", just replace the eval function with a call to console.log for example, and you will have your original code. Jsnice doesn't exceutes eval as it's doing statical analysis ie. without executing the programs.
- FiloSottile 12y ago
  The various jsBeautifiers special-case the eval() obfuscations, and in particular packer. I think the scope of this tool is to annotate and de-uglify js code without changing the logic, so you get a more readable version of the routine that generates the eval()'d code, that sounds right to me. EDIT: btw, can you please put the code in a code box, it messes with the layout. Mods: this is happening frequently, is it a bug?
FeatureRush 12y ago
Can this approach be extended to for example generating "matching" tests for the code? Like "I see this function processes dates, here are some popular test cases learned from 1000s other projects"? Could someone point me to good resources about mining code, most data mining and machine learning articles deal with points in multidimensional space and not objects with complex internal structure like programs...
- infogulch 12y ago
  The complex internal structure of programs could be reduced to graphs, and machine learning has been working with graphs for decades.
tantalor 12y ago
Why was it unable to infer the type of the return value for generateSeries? The function only has one return statement, and it already knows the type of the return variable is {Array}.
- veselin 12y ago
  It never shows types of return values, unless it is undefined. The reason is that the method may be overridden with a method with other return type.
nolanpro 12y ago
What is this sorcery?!?!? Infinitely helpful in reverse engineering google stuff like www.googletagservices.com/tag/js/gpt.js Thanks!!!
joshribakoff 12y ago
I put in some real world JS found on Hulu, and got a slew of errors like this one: Line 1: Parse error. missing ; before statement
- mvechev 12y ago
  Josh, Thanks for trying it out. I tried few large samples from Hulu and they seemed to work fine, e.g.: http://static.huluim.com/huluguru/i18n/en-us/translations-9d4d6e9074fa024132b56b9ce3aeee71.js http://static.huluim.com/huluguru/i18n/en-us/translations-9d... But indeed, sometimes there could be issues if the code does not compile with the compiler of choice.
Groxx 12y ago
Some rather spectacular failures, though of course these things happen with statistical methods: /** * @param {string} val * @return {?} */ var expect = function(val) { return "string" == typeof val; }; /** * @param {boolean} deepDataAndEvents * @return {?} */ var clone = function(deepDataAndEvents) { return "boolean" == typeof deepDataAndEvents; }; /** * @param {(boolean|number|string)} obj * @return {?} */ var isString = function(obj) { return "number" == typeof obj; }; Still, neat idea. Seems like there's a lot of room to train it, probably a lot of fun to try to improve things :)
dblotsky 12y ago
I wonder how many people, like me, right off the bad went and pasted minified jQuery into this thing. Looks like an invaluable tool for reverse-engineering when unminified code is unavailable.
enixn 12y ago
So, I saw this and wanted to see if it could decode output of something I saw earlier: http://patriciopalladino.com/files/hieroglyphy/ http://patriciopalladino.com/files/hieroglyphy/ I took the source of a sample pasted here and ran it through the heiroglyphy generator: var expect = function(val) { return "string" == typeof val; }; which output something along the lines of (truncated): [][(![]+[])[!+[]+!![]+!![]]+([]+{})[+!![]]+(!![]+[])[+!![]]+(!![]+[])[+[]]][([]+{})[!+[]+!![]+!![]+!![]+!![]]+([]+{})... But JSNice was unable to deobfuscate this code. Any ideas why?
- NaNaN 12y ago
  This involves program optimizations. JSNice doesn't optimize `var x = 1+1` to `var x = 2`
enscr 12y ago
Is it doing something beyond pretty print in Chrome?
- placeybordeaux 12y ago
  Yes.
insyncim64 12y ago
Nice tool!
SimeVidas 12y ago
> var width = $container.height(); Hehe

173 points by mvechev 12y ago

Whoa, it even infers local variable names automatically.

About: http://www.srl.inf.ethz.ch/jsnice.php http://www.srl.inf.ethz.ch/jsnice.php Apparently this is a machine learning research project. It seems that this would be quite useful for looking at the code of closed-source webapps. Would love to see it open-sourced or available as a service (via an API perhaps). Think auto-deminifying browser extension.

geetee 12y ago
On average, more than 60% of the identifiers are recovered to the same name as before the minification process. I wonder how much of that 60% comes from common libraries like jQuery or Underscore. Still, a neat project.
- veselin 12y ago
  We tried to evaluate on data that is as much as possible independent from the training data. So we evaluate on projects outside of github.

leeoniya 12y ago

didn't do too well on http://js1k.com/2014-dragons/details/1903 http://js1k.com/2014-dragons/details/1903 :) but processed jQuery relatively fast.

veselin 12y ago

Yes, it is statistical. A browser extension would be cool!

yconst 12y ago

Very interesting project. Would be great to get some more info on the actual algorithms being used. The about page offers relatively limited info.

mvechev 12y ago

Thank you all for the comments...keep them coming. we will definitely soon provide more details on how the overall system works...

pbielik 12y ago

Handy tool to have. Works suprisingly good.

mallamanis 12y ago

Naturalize ( http://groups.inf.ed.ac.uk/naturalize/ http://groups.inf.ed.ac.uk/naturalize/ ) is another related machine learning based tool that suggests appropriate identifier names for Java. Details on the implementation and an evaluation may be found here http://arxiv.org/abs/1402.4182 http://arxiv.org/abs/1402.4182

pajtai 12y ago

I took the code presented and put it into packer - http://dean.edwards.name/packer/ http://dean.edwards.name/packer/ , and the "nice" output was not very helpful in that it still looked obfuscated. Maybe I'm misunderstanding something. So the input to js nice was the packed generateSeries function: eval(function(p,a,c,k,e,r){e=function(c) {return(c<a?'':e(parseInt(c/a)))+((c=c%a)>35? String.fromCharCode(c+29):c.toString(36))};if(!''.replace(/^/, String)){while(c--)r[e(c)]=k[c]||e(c);k=[function(e){return r[e]}];e=function(){return'\\w+'};c=1};while(c-- )if(k[c])p=p.replace(new RegExp('\\b'+e(c)+'\\b','g'),k[c]);return p}('4 3(e){2 t=[];2 n=f+e;2 r=6+e;k(i=1;i<=6;i++){2 s=5.y(5.a()*(r- n+1)+n);t.b([i,s]);n++;r++}c t}$(d).B(4(){2 e=3(0);2 t=3(g);$.h($("#j"),[{7:"l",8:e},{7:"o",8:t}],{p:{q:{u: ["#v","#w"]}},x:{9:z},A: {9:m}})})',38,38,'||var|generateSeries|function|Math|200|label |data|ticks|random|push|return|document||100|300|plot||flotcon tainer|for|data1|10||data2|grid|backgroundColor||||colors|D1D1 D1|7A7A7A|xaxis|floor|20|yaxis|ready'.split('|'),0,{}))

mVChr 12y ago
Noticed that too, was kind of happy about it. For most projects I use uglify or Google Closure Compiler and the like, which it seems to do a good job with. When I really want something obfuscated to make it as difficult as possible for someone to figure out what's going on behind the scenes though I use packer and am glad this tool couldn't figure out what to do with it.
spicyj 12y ago
http://jsbeautifier.org/ http://jsbeautifier.org/ has no problem deobfuscating that.
adrenalinup 12y ago
It's easy to depack even the packer's "Base 64 encode", just replace the eval function with a call to console.log for example, and you will have your original code. Jsnice doesn't exceutes eval as it's doing statical analysis ie. without executing the programs.
FiloSottile 12y ago
The various jsBeautifiers special-case the eval() obfuscations, and in particular packer. I think the scope of this tool is to annotate and de-uglify js code without changing the logic, so you get a more readable version of the routine that generates the eval()'d code, that sounds right to me. EDIT: btw, can you please put the code in a code box, it messes with the layout. Mods: this is happening frequently, is it a bug?

FeatureRush 12y ago

Can this approach be extended to for example generating "matching" tests for the code? Like "I see this function processes dates, here are some popular test cases learned from 1000s other projects"? Could someone point me to good resources about mining code, most data mining and machine learning articles deal with points in multidimensional space and not objects with complex internal structure like programs...

infogulch 12y ago
The complex internal structure of programs could be reduced to graphs, and machine learning has been working with graphs for decades.

tantalor 12y ago

Why was it unable to infer the type of the return value for generateSeries? The function only has one return statement, and it already knows the type of the return variable is {Array}.

veselin 12y ago
It never shows types of return values, unless it is undefined. The reason is that the method may be overridden with a method with other return type.

nolanpro 12y ago

What is this sorcery?!?!? Infinitely helpful in reverse engineering google stuff like www.googletagservices.com/tag/js/gpt.js Thanks!!!

joshribakoff 12y ago

I put in some real world JS found on Hulu, and got a slew of errors like this one: Line 1: Parse error. missing ; before statement

mvechev 12y ago
Josh, Thanks for trying it out. I tried few large samples from Hulu and they seemed to work fine, e.g.: http://static.huluim.com/huluguru/i18n/en-us/translations-9d4d6e9074fa024132b56b9ce3aeee71.js http://static.huluim.com/huluguru/i18n/en-us/translations-9d... But indeed, sometimes there could be issues if the code does not compile with the compiler of choice.

Groxx 12y ago

Some rather spectacular failures, though of course these things happen with statistical methods: /** * @param {string} val * @return {?} */ var expect = function(val) { return "string" == typeof val; }; /** * @param {boolean} deepDataAndEvents * @return {?} */ var clone = function(deepDataAndEvents) { return "boolean" == typeof deepDataAndEvents; }; /** * @param {(boolean|number|string)} obj * @return {?} */ var isString = function(obj) { return "number" == typeof obj; }; Still, neat idea. Seems like there's a lot of room to train it, probably a lot of fun to try to improve things :)

dblotsky 12y ago

I wonder how many people, like me, right off the bad went and pasted minified jQuery into this thing. Looks like an invaluable tool for reverse-engineering when unminified code is unavailable.

enixn 12y ago

So, I saw this and wanted to see if it could decode output of something I saw earlier: http://patriciopalladino.com/files/hieroglyphy/ http://patriciopalladino.com/files/hieroglyphy/ I took the source of a sample pasted here and ran it through the heiroglyphy generator: var expect = function(val) { return "string" == typeof val; }; which output something along the lines of (truncated): [][(![]+[])[!+[]+!![]+!![]]+([]+{})[+!![]]+(!![]+[])[+!![]]+(!![]+[])[+[]]][([]+{})[!+[]+!![]+!![]+!![]+!![]]+([]+{})... But JSNice was unable to deobfuscate this code. Any ideas why?

NaNaN 12y ago
This involves program optimizations. JSNice doesn't optimize `var x = 1+1` to `var x = 2`

enscr 12y ago

Is it doing something beyond pretty print in Chrome?

placeybordeaux 12y ago
Yes.

insyncim64 12y ago

Nice tool!

SimeVidas 12y ago

> var width = $container.height(); Hehe