3 ms·

Thank you! We have a common AST format we use as the representation of every language we support. You add new languages to Optic by writing a mapper from their

by addcn 8y ago

Thank you! We have a common AST format we use as the representation of every language we support. You add new languages to Optic by writing a mapper from their AST to ours. It should work with pretty much any context-free AST. Here's what that looks like for JS https://github.com/opticdev/es7-parser/blob/master/src/main/scala-2.12/com/opticdev/parsers/es7/ASTJsonToGraph.scala https://github.com/opticdev/es7-parser/blob/master/src/main/... We can't translate code between languages but we do the next best thing by moving the meaning of that code around. ie backend is Flask and frontend is JS. We can generate JS code that connects to the Flask API automatically. Graal is pretty cool, and I'm happy to see a big org backing a polygot tool. I'm most interested in Truffle and looking into ways it can make Optic better. Have any ideas? Thanks for catching the 404! I'll look into it ASAP

sgrove 8y ago
How much work is it to add a new language if you have to get it into your own format? We use some esoteric languages (Clojure and Reason/OCaml) so I don't expect Optic to support our use case any time soon, but I'd be curious for e.g. Python/Ruby. Also, a couple of other question that come to mind: 1. Jared Forsyth gave a great talk on avoiding breaking changes as a library author by using code-mods - would Optic be able to wrap up some of the changes it makes to my library code as a code-mod that I could ship with a new version of my code to auto-upgrade all my dependents? That could significantly help with some of the higher-churn libraries (some people are iffy about this though). 2. Have you considered using something closer to lsp for the editor protocol? Emacs support would be nice, and that'd be a good way to jumpstart the implementation. 3. It actually doesn't look too bad to implement the plugin for e.g. even without lsp (https://useoptic.com/docs/authoring/adding-optic-support-for-new-id-es https://useoptic.com/docs/authoring/adding-optic-support-for...). Looking at the 'search' event though, why use `///` instead of picking up `cmd-f` (or `C-s` in emacs)? I was a bit confused about the interaction model. Optic looks pretty interesting, best of luck!
- addcn 8y ago
  All you have to do is pick an existing parser that works well and map the output AST to Optic's AST. I'm exploring building an Antlr extension so any grammar in their library could output in Optic's AST but that's still not implemented. Every AST have a notion of type, range, and properties which can either be primitives, arrays of child nodes, or individual child nodes. We have Python and Ruby on our roadmap for the next couple of weeks. 1. We've been exploring migrating one Optic skill to a newer version. So far it seems promising for the cases you'd expect a code-mod to help with, but if you completely change the API and deprecate some stuff it can point out the changes but won't be able to help you -- no magic here. 2. I looked into language servers (lsp) but the implementation was a lot more involved and I found bugs in some of the IDEs that claim to have full lsp support. As the protocol matures and we add more functionality to Optic I expect to see lsp become our main avenue for IDE/Optic interactions. 3. We wanted to make it really natural to ask Optic a question. We thought '///' would be unique and easy to remember, but we're open to changing it. It's nice that '///' also evaluates as a comment too because before we used it sometimes characters in your search would cause the AST parse to fail. We're open to any ideas here -- a lot of users have said it's funky.
carapace 8y ago
Two things: first, it's very impressive that you're doing the hard work to integrate this with IDEs and such. To me that shows that you're really serious about getting real-world uptake. There are a few tools and systems that do what you're doing but most of them never seem to make it out of some tiny niche and remain academic. Second, your Marvin sub-project, if I understand it correctly, is just awesome. Congratulations. In re: Truffle, and lang-to-AST conversion in general, I've taken a winding path over the better part of the last twenty-odd years. The closest thing to a serious idea I have to offer is this: Use Lisp as your inter-language AST. I'm grinning but I'm not quite joking. ;-) Think it through. What are the pros and cons? (I can't help but mention, I'm working on a system that may one day "eat" other software and translate it into essentially an AST. The AST is actually just code in a language called Joy that resembles Forth+Lisp. The language is very simple and can be implemented on top of other languages in a few days. So I can (eventually) read in other code to this Joy/AST, clean it up, then run it on whatever substratum makes sense for a given use-case. Joy code is easily refactored, and can be fairly easily converted to logical relations and then you can use e.g. Prolog or miniKanren to search for optimizations. This is sort of "one step beyond" type inference, moving into the more esoteric but useful things where you can do partial evaluation and "super-compilation" (chase down alternate control-flow paths) to improve the abstract code and emit specialized e.g. assembly or LLVM IR or JVM bytecode or whatever. My motivations are two: I want to teach normal people to program, and I want to contract the total amount of [distinct] software in the world, eliminating incorrect code and collecting, refining, and curating all correct code into a single unified codebase. The Joy notation is easy enough to teach to normal people. I've created a simple GUI that presents a Joy environment where users can build new commands (Joy code). The system uses type inference to prevent the user from executing commands that won't work with the current system state, and to prevent the construction of new commands that won't work at all. It's impossible to make incorrect code in this (virtual) world. The Joy language is highly-factorable, enabling a kind of "semantic compression". It's also Categorical (in the sense of Category Theory), meaning that programs in Joy represent abstract computations over categories and can deliver multiple correct programs for the same Joy expression. For example, the type inferencer works by evaluating Joy expressions over a category of stack effects and type variables to generate the stack effects of the expressions. Conal Elliot has a paper "Compiling to Categories" where he's exploring this sort of thing using Haskell, converting it to a "point-free" Joy-like notation and then implementing categories for dataflow diagrams and circuit descriptions and other cool things, to produce those all from the same original Haskell code. I'm hopeful that I've got something here. :-) I've been sitting in a room in front of a computer for about a year and I'm just at the point where I have to go out and talk to people about it. Egad. Anyhow, that's part of why I'm so impressed with what you've accomplished (and are no doubt going to accomplish!) I look at what you've done already and I think, "Daaaamn, this dude is running circles around me." Makes me feel sheepish, but I'm stoked for you! You're bringing the fire down from Olympus my Promethean friend, don't doubt it.

by addcn 8y ago

sgrove 8y ago

How much work is it to add a new language if you have to get it into your own format? We use some esoteric languages (Clojure and Reason/OCaml) so I don't expect Optic to support our use case any time soon, but I'd be curious for e.g. Python/Ruby. Also, a couple of other question that come to mind: 1. Jared Forsyth gave a great talk on avoiding breaking changes as a library author by using code-mods - would Optic be able to wrap up some of the changes it makes to my library code as a code-mod that I could ship with a new version of my code to auto-upgrade all my dependents? That could significantly help with some of the higher-churn libraries (some people are iffy about this though). 2. Have you considered using something closer to lsp for the editor protocol? Emacs support would be nice, and that'd be a good way to jumpstart the implementation. 3. It actually doesn't look too bad to implement the plugin for e.g. even without lsp (https://useoptic.com/docs/authoring/adding-optic-support-for-new-id-es https://useoptic.com/docs/authoring/adding-optic-support-for...). Looking at the 'search' event though, why use `///` instead of picking up `cmd-f` (or `C-s` in emacs)? I was a bit confused about the interaction model. Optic looks pretty interesting, best of luck!

addcn 8y ago
All you have to do is pick an existing parser that works well and map the output AST to Optic's AST. I'm exploring building an Antlr extension so any grammar in their library could output in Optic's AST but that's still not implemented. Every AST have a notion of type, range, and properties which can either be primitives, arrays of child nodes, or individual child nodes. We have Python and Ruby on our roadmap for the next couple of weeks. 1. We've been exploring migrating one Optic skill to a newer version. So far it seems promising for the cases you'd expect a code-mod to help with, but if you completely change the API and deprecate some stuff it can point out the changes but won't be able to help you -- no magic here. 2. I looked into language servers (lsp) but the implementation was a lot more involved and I found bugs in some of the IDEs that claim to have full lsp support. As the protocol matures and we add more functionality to Optic I expect to see lsp become our main avenue for IDE/Optic interactions. 3. We wanted to make it really natural to ask Optic a question. We thought '///' would be unique and easy to remember, but we're open to changing it. It's nice that '///' also evaluates as a comment too because before we used it sometimes characters in your search would cause the AST parse to fail. We're open to any ideas here -- a lot of users have said it's funky.

carapace 8y ago

Two things: first, it's very impressive that you're doing the hard work to integrate this with IDEs and such. To me that shows that you're really serious about getting real-world uptake. There are a few tools and systems that do what you're doing but most of them never seem to make it out of some tiny niche and remain academic. Second, your Marvin sub-project, if I understand it correctly, is just awesome. Congratulations. In re: Truffle, and lang-to-AST conversion in general, I've taken a winding path over the better part of the last twenty-odd years. The closest thing to a serious idea I have to offer is this: Use Lisp as your inter-language AST. I'm grinning but I'm not quite joking. ;-) Think it through. What are the pros and cons? (I can't help but mention, I'm working on a system that may one day "eat" other software and translate it into essentially an AST. The AST is actually just code in a language called Joy that resembles Forth+Lisp. The language is very simple and can be implemented on top of other languages in a few days. So I can (eventually) read in other code to this Joy/AST, clean it up, then run it on whatever substratum makes sense for a given use-case. Joy code is easily refactored, and can be fairly easily converted to logical relations and then you can use e.g. Prolog or miniKanren to search for optimizations. This is sort of "one step beyond" type inference, moving into the more esoteric but useful things where you can do partial evaluation and "super-compilation" (chase down alternate control-flow paths) to improve the abstract code and emit specialized e.g. assembly or LLVM IR or JVM bytecode or whatever. My motivations are two: I want to teach normal people to program, and I want to contract the total amount of [distinct] software in the world, eliminating incorrect code and collecting, refining, and curating all correct code into a single unified codebase. The Joy notation is easy enough to teach to normal people. I've created a simple GUI that presents a Joy environment where users can build new commands (Joy code). The system uses type inference to prevent the user from executing commands that won't work with the current system state, and to prevent the construction of new commands that won't work at all. It's impossible to make incorrect code in this (virtual) world. The Joy language is highly-factorable, enabling a kind of "semantic compression". It's also Categorical (in the sense of Category Theory), meaning that programs in Joy represent abstract computations over categories and can deliver multiple correct programs for the same Joy expression. For example, the type inferencer works by evaluating Joy expressions over a category of stack effects and type variables to generate the stack effects of the expressions. Conal Elliot has a paper "Compiling to Categories" where he's exploring this sort of thing using Haskell, converting it to a "point-free" Joy-like notation and then implementing categories for dataflow diagrams and circuit descriptions and other cool things, to produce those all from the same original Haskell code. I'm hopeful that I've got something here. :-) I've been sitting in a room in front of a computer for about a year and I'm just at the point where I have to go out and talk to people about it. Egad. Anyhow, that's part of why I'm so impressed with what you've accomplished (and are no doubt going to accomplish!) I look at what you've done already and I think, "Daaaamn, this dude is running circles around me." Makes me feel sheepish, but I'm stoked for you! You're bringing the fire down from Olympus my Promethean friend, don't doubt it.