4 ms·
There's nothing "wrong" with the Java or V8 approach per se. It's not even a given that SDE would do better today. Note that most of Franz' own work has since b
by vidarh 9y ago
There's nothing "wrong" with the Java or V8 approach per se. It's not even a given that SDE would do better today. Note that most of Franz' own work has since been on the JVM.
I just think SDE is worth more exploration. It lost out not because it failed to compete on merits with the JVM but because the JVM basically cut short any hope of it getting mindshare at the time.
It's possible the JVM or V8 approaches are simply superior designs. But we don't really know, because such huge resources have gone into making the JVM or v8 approaches work, and not just SDE but most other approaches from that time became near instant dead ends.
> The v8 / "modern JS" approach is to do gzip compression of source code on one side of the network. Then decompression, lazily parse, and lazily generate native code on the other side.
The interesting part of this is that you could treat SDE simply as a way to short-circuit the parsing and compression step by using it to serialize the AST and still retain exactly the same code generation. You can lazily generate native code from it, as long as you apply the heuristics to build the dictionary on load time (necessary to rebuild the correct tree).
Whether it'd be worth it is an open question, but I wish I had time to explore it.
- vanderZwan 9y agoI'm actually exploring something "spiritually" related: currently, I have a project where I store the state of an app in the URL, by compressing trees of JavaScript objects by first putting them through a pre-compression "filter" step to make the data more compressible (not going into that now, but see [0]), then apply JSON.stringify, and finally put the resulting string through lz-string to create an URI-safe string[1]. lz-string is basically a wonderful, wonderful hack to Lempel-Ziv compress strings to a bitstream, which is then turned back into a string. I'm basically looking at ways to cut out the JSON.stringify step. So basically, first turn a JSON-friendly object into a binary representation, which is then compressed into a bytestream, and stored in a URI-safe string. This might lead to something generally usable, and faster/smaller than lz-string. I know JSON alternatives like flatpack and messagepack exist, but this would be a much simpler, lighter JS library, single-digit kilobytes in size before gzipping. It also would probably be backwards compatible all the way to IE6. [0] https://github.com/linnarsson-lab/loom-viewer/blob/master/client/js/state-compressor.js https://github.com/linnarsson-lab/loom-viewer/blob/master/cl... [1] http://pieroxy.net/blog/pages/lz-string/index.html http://pieroxy.net/blog/pages/lz-string/index.html
- vidarh 9y agoThat's interesting. Yes, I'd encourage looking at how SDE functions there, as SDE is ultimately using Lempel-Ziv-Welch that operates directly on a tree instead of a serialized string, and where the heuristics for what to add to the dictionary can include templated tree subsets. That part of it is really independent of the code generation aspect, and could easily be adapted to serialize arbitrary types of tree structures. Basically what you need is a general mechanism that takes a possibly templated rule, and adds to the dictionary. Then you need a mechanism that looks at a JS object and instead of spitting out JSON, matches the current node to one of the dictionary items and spits out a bit pattern and then recursively spits out the bit pattern for the template arguments. The decoder can basically be very simple recursive descent: Identify the dictionary entry, look it up, find the template rule, recursively call itself to retrieve the template patterns, and reconstitute the object. The "magic" then lies in the heuristics, which can be very generic (e.g. suitable for any JSON) or very specific (e.g. embedding domain knowledge of what the trees are likely to be shaped like, or at least code to be able to recognize and encode complex shapes when seen).
- chubot 9y agoOK, fair enough! FWIW here is a long thread I started about an AST serialization method, with some prototype code: https://www.reddit.com/r/ProgrammingLanguages/comments/79fkpu/representing_asts_as_byte_strings_with_with_small/ https://www.reddit.com/r/ProgrammingLanguages/comments/79fkp... It is lightly "compressed" -- there are no native pointers, but offsets instead. The in-memory format is the same as the on-disk format. You can mmap() a file and then start following "pointers" right away. I didn't develop it for transmission over a network, but it could be used in place of Python's .pyc files, which essentially cache parsing and bytecode compilation, and are a big tree/graph data structure. It can represent arbitrary graphs. You could add dictionary compression in a tree-aware way or a tree-oblivious way... not sure how that compares to SDE. FWIW someone suggested WebAssembly on that thread, but it's apparently no longer an AST -- it's bytecode (I guess more similar to Java.)
- vidarh 9y agoThat's interesting. I think where SDE goes further is that you can go full lzw with it. The "dictionary" here is not a static dict, but a dynamically built dictionary as with lzw compression that is expanded as you parse the stream. That makes it harder to traverse - you can't jump to arbitrary locations without decoding everything before it, because you need to look at everything to apply the heuristics to build the dictionary. But it makes it likely to be far more compressed - e.g. you can have dictionary entries shorter than a byte, and you can build dictionary entries that represents whole subtrees, and in SDE's case those subtrees can be templated.
- chubot 9y agoHm, but if compression ratio is the only difference, then I don't see that as a bottleneck in most language systems? I think that overall startup time for example is a more interesting goal, not just the amount of data transferred over the network or streamed off disk. Startup time can be saved by doing parsing and semantic analysis on one side of the network. And if you know your target architecture, you can do code generation there too, although that doesn't seem to be the way any systems have evolved -- even Android was doing native code generation on the device recently. I guess I am having trouble seeing situations where both of these are true: 1) A compression algorithm specific to a file format is say 50% or 100% better than gzip or xz. 2) That 50% or 100% results in a big user-perceived improvement. Most bytecode formats are "lazily" decoded, although they can be somewhat large. I sort of like the scheme I proposed because it's both compact and can be lazily decoded. You could make it more compact, but then you would have to decompress the whole thing up front.