4 ms·

And badly. I rrally wish JavaScript would just use DWARF. Sourcemaps are quite broken, do not contain scope infos and cannot be used to map function names wirho

And badly. I rrally wish JavaScript would just use DWARF. Sourcemaps are quite broken, do not contain scope infos and cannot be used to map function names wirhout doing stupid things.

khuey 9y ago
DWARF is kind of a shitshow and it's not really suited for doing this sort of source to source transformation anyways.
- the_mitsuhiko 9y ago
  I do not see why not. You can compile javascript into a single line and then use the column index as memory address and dwarf should work almost unchanged. Also not sure why DWARF is a shitshow.
  - Peaker 9y ago
    DWARF is terribly designed. It is far more complicated than it needs to be. For example, to map binary locations to line numbers -- it uses its own opcodes and registers in a custom VIRTUAL MACHINE(!!) When it could have had a simple table like: Line Offset 1 0xa 2 0x14 4 0x2a ... Then just compress that with an ordinary compression algorithm rather than inventing silly virtual machines. The debug info itself is encoded in a very cumbersome encoding of debug-information-entries that point to one another. A much simpler method would have been RLP, for example.
    - the_mitsuhiko 9y ago
      The line number programs of DWARF are pretty well working for the problems they should solve. Out of all things in DWARF that are weird, these are not the issue. We use DWARF at Sentry just fine and I am quite a big of a supporter of the format as ypu can guess. In particular it was designed and specified unlike sourcemaps which are a random google docs page and don’t even solve basic problems such as finding out which function a token belongs to.
    - Jasper_ 9y ago
      DWARF solves a much harder problem than mapping binary offsets to line numbers. DWARF solves the problem of transforming optimized code into something debuggable. For instance: https://godbolt.org/g/cHvnCh https://godbolt.org/g/cHvnCh This was compiled with debug info turned on. Where did the memcpy line go? Turns out, it can be optimized out entirely and the whole function combined. There's no possible mapping back to source code. DWARF allows you to transform this back to stupider, "unoptimized code", which can be stepped through line-by-line. This is one basic reason why Source Maps are useless: in anything but the stupidest compilers, transformation back to the source code is more complicated than table mapping.
      - Peaker 9y ago
        Whatever information is within the line number information - it can stored in a simple, naive way - and then compressed normally. There is absolutely no justification in inventing a virtual machine with its own opcodes for this.
        the_mitsuhiko 9y ago
        >Whatever information is within the line number information - it can stored in a simple, naive way - and then compressed normally. Not sure how a compression is going to be better than a VM. The "vm" here is super simple and achieves significantly better compression than an actual compression algorithm. And it's easier to implement and work with. Also again this is not just line information so you really want a state machine for this or this explodes in more and more complexity. We built a system that generates out simple mappings from DWARF's line number programs to files we can mmap and it's only smaller for the case we are about (line number info). Anything else and DWARF's programs are better. So no surprised DWARF works the way it does.
        Peaker 9y ago
        When you just want line number info from DWARF -- all the existing tools are extremely slow. A simple sorted address->line table with binary search is incredibly faster. This is a very common use case. At the very least, this proves DWARF is not designed for its common use cases, not properly at least.
cromwellian 9y ago
JavaScript doesn't use DWARF. DevTools and production server systems do. If you think DWARF is better, write up a patch to Babel or Closure Compiler, and a devtools extension. I think you'll problem find that DWARF doesn't really match the source-to-source profile case well, and in the end, you'll add so many extensions that what you end up with isn't DWARF at all. SourceMaps can be used to global names, but not locals. However, Closure Compiler has the ability to generate more elaborate maps that do allow reverse mapping. This is how Google services deobfuscate logs and stack traces sent back by heavily optimized clients.
- the_mitsuhiko 9y ago
  > However, Closure Compiler has the ability to generate more elaborate maps that do allow reverse mapping Is that an extension to source maps that Google came up with? Because from the current specification I do not see how this can be done without hacks. The way we're doing it is inverse token search from a starting position over the minified tokens until we find our minified function name followed by the keyword 'function'. And that approach is slow and not entirely correct.
  - cromwellian 9y ago
    SourceMaps as specified are an interop format for devtools on the client side. On the server side, you can store a lot more information since it doesn't need to be transmitted to the client. Closure Compiler stores maps for all variables, all properties, all functions, all renamed strings, all idGenerators, etc. You can store these if you want. Google's servers store these maps and when user feedback or exceptions are logged, they are used to deobfuscate them. SourceMaps + functionMaps + propertyMaps + the others I mentioned as used to deobfuscate. This doesn't solve the problem of deobfuscating locals or heap objects. That needs an extension.
    - the_mitsuhiko 9y ago
      Sure. But what you describe is not possible with the documented sourcemap format. You can’t even map function names at the moment. It’s purely a token and token location mapping format.

And badly. I rrally wish JavaScript would just use DWARF. Sourcemaps are quite broken, do not contain scope infos and cannot be used to map function names wirhout doing stupid things.

khuey 9y ago

DWARF is kind of a shitshow and it's not really suited for doing this sort of source to source transformation anyways.

the_mitsuhiko 9y ago
I do not see why not. You can compile javascript into a single line and then use the column index as memory address and dwarf should work almost unchanged. Also not sure why DWARF is a shitshow.
- Peaker 9y ago
  DWARF is terribly designed. It is far more complicated than it needs to be. For example, to map binary locations to line numbers -- it uses its own opcodes and registers in a custom VIRTUAL MACHINE(!!) When it could have had a simple table like: Line Offset 1 0xa 2 0x14 4 0x2a ... Then just compress that with an ordinary compression algorithm rather than inventing silly virtual machines. The debug info itself is encoded in a very cumbersome encoding of debug-information-entries that point to one another. A much simpler method would have been RLP, for example.
  - the_mitsuhiko 9y ago
    The line number programs of DWARF are pretty well working for the problems they should solve. Out of all things in DWARF that are weird, these are not the issue. We use DWARF at Sentry just fine and I am quite a big of a supporter of the format as ypu can guess. In particular it was designed and specified unlike sourcemaps which are a random google docs page and don’t even solve basic problems such as finding out which function a token belongs to.
  - Jasper_ 9y ago
    DWARF solves a much harder problem than mapping binary offsets to line numbers. DWARF solves the problem of transforming optimized code into something debuggable. For instance: https://godbolt.org/g/cHvnCh https://godbolt.org/g/cHvnCh This was compiled with debug info turned on. Where did the memcpy line go? Turns out, it can be optimized out entirely and the whole function combined. There's no possible mapping back to source code. DWARF allows you to transform this back to stupider, "unoptimized code", which can be stepped through line-by-line. This is one basic reason why Source Maps are useless: in anything but the stupidest compilers, transformation back to the source code is more complicated than table mapping.
    - Peaker 9y ago
      Whatever information is within the line number information - it can stored in a simple, naive way - and then compressed normally. There is absolutely no justification in inventing a virtual machine with its own opcodes for this.
      - the_mitsuhiko 9y ago
        >Whatever information is within the line number information - it can stored in a simple, naive way - and then compressed normally. Not sure how a compression is going to be better than a VM. The "vm" here is super simple and achieves significantly better compression than an actual compression algorithm. And it's easier to implement and work with. Also again this is not just line information so you really want a state machine for this or this explodes in more and more complexity. We built a system that generates out simple mappings from DWARF's line number programs to files we can mmap and it's only smaller for the case we are about (line number info). Anything else and DWARF's programs are better. So no surprised DWARF works the way it does.
        Peaker 9y ago
        When you just want line number info from DWARF -- all the existing tools are extremely slow. A simple sorted address->line table with binary search is incredibly faster. This is a very common use case. At the very least, this proves DWARF is not designed for its common use cases, not properly at least.

cromwellian 9y ago

JavaScript doesn't use DWARF. DevTools and production server systems do. If you think DWARF is better, write up a patch to Babel or Closure Compiler, and a devtools extension. I think you'll problem find that DWARF doesn't really match the source-to-source profile case well, and in the end, you'll add so many extensions that what you end up with isn't DWARF at all. SourceMaps can be used to global names, but not locals. However, Closure Compiler has the ability to generate more elaborate maps that do allow reverse mapping. This is how Google services deobfuscate logs and stack traces sent back by heavily optimized clients.

the_mitsuhiko 9y ago
> However, Closure Compiler has the ability to generate more elaborate maps that do allow reverse mapping Is that an extension to source maps that Google came up with? Because from the current specification I do not see how this can be done without hacks. The way we're doing it is inverse token search from a starting position over the minified tokens until we find our minified function name followed by the keyword 'function'. And that approach is slow and not entirely correct.
- cromwellian 9y ago
  SourceMaps as specified are an interop format for devtools on the client side. On the server side, you can store a lot more information since it doesn't need to be transmitted to the client. Closure Compiler stores maps for all variables, all properties, all functions, all renamed strings, all idGenerators, etc. You can store these if you want. Google's servers store these maps and when user feedback or exceptions are logged, they are used to deobfuscate them. SourceMaps + functionMaps + propertyMaps + the others I mentioned as used to deobfuscate. This doesn't solve the problem of deobfuscating locals or heap objects. That needs an extension.
  - the_mitsuhiko 9y ago
    Sure. But what you describe is not possible with the documented sourcemap format. You can’t even map function names at the moment. It’s purely a token and token location mapping format.