4 ms·

This seems to be advocating to turn program programming into compiler programming. Like it took "every sufficiently complicated program reimplements half of lis

This seems to be advocating to turn program programming into compiler programming. Like it took "every sufficiently complicated program reimplements half of lisp" and said that was an admirable goal and here's how to do it well. I only skimmed the initial algebra talk linked out, but it seems to mirror what any given programming language does with your code. It takes your text, does some processing to turn it into an AST, then turns that into a compiled program and runs it. AFAICT the initial algebra suggestion says to basically program directly with your AST wherever you can, and then write code to turn that into a program. Now you've written an AST and a compiler, but naively interpreting that AST will probably be slow, so maybe you work in some optimizations. And you don't always write correct code, so you have to add a debugger too. Now you've just written a programming language and toolset. Why not just use an existing language and mature toolset? Is it because your DSL might be constrained to the original problem well enough to be simpler than anything general purpose? Replacing e.g. `map` and a native function with `MyMap` and a few of my own functions seems to be throwing out all the good that comes with a mature language ecosystem.

Smaug123 7y ago
Yes, I think you've basically summed it up very well! One of the reasons we use this pattern internally is because we have quite specific performance requirements which are not well-served by your standard language compiler/runtime. If we handle some of this compilation workload, we can make sure we emit constructs in the underlying runtime which have the right performance properties. We also want to make sure our users don't really need to think about this sort of low-level mucking around with performance; so we use the initial algebra to expose natural data-driven abstractions to them, which we then carefully manipulate into the right forms for the .NET runtime to have our desired performance properties. This is an area where F# really shines. Its "computation expressions" make it really easy to construct DSLs embedded in F#. We offer our users this DSL, and we promise that anything written using this DSL will be appropriately fast and safe; but we do sometimes give them an escape hatch. By encouraging the user to stick to this heavily restricted DSL, we make it easier for them to write code that we guarantee will perform well. If the code doesn't perform well, that's a big problem, but crucially it's a problem for the devs to solve, not for the quant researchers to waste hours digging into. Entirely separately from the above, we can also offer certain safety guarantees by restricting to our DSLs. One project in particular has involved taking something that previously existed but required a lot of manual procedural bookkeeping on the part of the user, and extracting the "intent" of the library in a purely data-oriented DSL. As long as the user sticks to our DSL, they don't need to consider how their computations are sequenced; we'll do that for them, in the process of converting from the DSL into the underlying system.
- 6gvONxR4sf7o 7y ago
  Are you effectively interpreting it, or are you compiling it to something else? I'm really curious how you do this in a performant way.
  - Smaug123 7y ago
    We have places where we ultimately emit IL (the bytecode of .NET). However, usually we just emit F#, restricted to certain constructs which are allocation-free and so forth. We aren't producing F# in the sense that we have to invoke the F# compiler ourselves, though. In the calculator example, given the description `Add (Negate (Const 5)) (Const 3)`, we might ultimately end up having assembled an F# function in memory like `fun () -> -5 + 3`; when this function is invoked, the value `-2` will be calculated. Ultimately we usually put together the functions in the usual way you make a function: by composition of smaller functions. The very simplest expressions like `Const 5` have a simple template like `fun () -> 5` which we can just directly produce; more complex expressions have to be interpreted recursively into F# function objects. I agree it's a bit hard to explain unless you're trying to solve a less trivial problem :(
- jhgb 7y ago
  > we have quite specific performance requirements which are not well-served by your standard language compiler/runtime. If we handle some of this compilation workload, we can make sure we emit constructs in the underlying runtime which have the right performance properties. Is it just me or does this sound a bit like Common Lisp's compiler macros (http://www.lispworks.com/documentation/HyperSpec/Body/03_bbaa.htm http://www.lispworks.com/documentation/HyperSpec/Body/03_bba...)?

Smaug123 7y ago

Yes, I think you've basically summed it up very well! One of the reasons we use this pattern internally is because we have quite specific performance requirements which are not well-served by your standard language compiler/runtime. If we handle some of this compilation workload, we can make sure we emit constructs in the underlying runtime which have the right performance properties. We also want to make sure our users don't really need to think about this sort of low-level mucking around with performance; so we use the initial algebra to expose natural data-driven abstractions to them, which we then carefully manipulate into the right forms for the .NET runtime to have our desired performance properties. This is an area where F# really shines. Its "computation expressions" make it really easy to construct DSLs embedded in F#. We offer our users this DSL, and we promise that anything written using this DSL will be appropriately fast and safe; but we do sometimes give them an escape hatch. By encouraging the user to stick to this heavily restricted DSL, we make it easier for them to write code that we guarantee will perform well. If the code doesn't perform well, that's a big problem, but crucially it's a problem for the devs to solve, not for the quant researchers to waste hours digging into. Entirely separately from the above, we can also offer certain safety guarantees by restricting to our DSLs. One project in particular has involved taking something that previously existed but required a lot of manual procedural bookkeeping on the part of the user, and extracting the "intent" of the library in a purely data-oriented DSL. As long as the user sticks to our DSL, they don't need to consider how their computations are sequenced; we'll do that for them, in the process of converting from the DSL into the underlying system.

6gvONxR4sf7o 7y ago
Are you effectively interpreting it, or are you compiling it to something else? I'm really curious how you do this in a performant way.
- Smaug123 7y ago
  We have places where we ultimately emit IL (the bytecode of .NET). However, usually we just emit F#, restricted to certain constructs which are allocation-free and so forth. We aren't producing F# in the sense that we have to invoke the F# compiler ourselves, though. In the calculator example, given the description `Add (Negate (Const 5)) (Const 3)`, we might ultimately end up having assembled an F# function in memory like `fun () -> -5 + 3`; when this function is invoked, the value `-2` will be calculated. Ultimately we usually put together the functions in the usual way you make a function: by composition of smaller functions. The very simplest expressions like `Const 5` have a simple template like `fun () -> 5` which we can just directly produce; more complex expressions have to be interpreted recursively into F# function objects. I agree it's a bit hard to explain unless you're trying to solve a less trivial problem :(
jhgb 7y ago
> we have quite specific performance requirements which are not well-served by your standard language compiler/runtime. If we handle some of this compilation workload, we can make sure we emit constructs in the underlying runtime which have the right performance properties. Is it just me or does this sound a bit like Common Lisp's compiler macros (http://www.lispworks.com/documentation/HyperSpec/Body/03_bbaa.htm http://www.lispworks.com/documentation/HyperSpec/Body/03_bba...)?