5 ms·

I started the hobbes project at MS and am happy to answer any questions that folks have about it!

ah- 9y ago
What motivated you to start building your own language? Did you use much kdb/q before? How did you manage to get this open sourced?
- throwaway7645 9y ago
  I'd like to know this as well. I know most array languages coupled with a DB aren't as fast as a language with a JIT, but the code is quite expressive. If Kdb+ can ever easily be run on an FPGA or GPU cluster...that would be something.
  - stuntprogrammer 9y ago
    Not kdb+, but a proprietary (internal-only) language that it heavily influenced was designed around execution on GPU clusters. The FPGAs were used mainly for feedhandlers and there was a different DSL for that (compiling to verilog). It was indeed rather something to see :)
    - throwaway7645 9y ago
      Was it received favorably? A secret sauce, or a failed project.
      - stuntprogrammer 9y ago
        Secret sauce, at least for the team involved, and well-received. Nice combination of extreme perf and decent productivity for them. I've moved into non-finance stuff for quite a while now though, so not sure what's become of it. Given business challenges in that particular sub-field, who knows..
- kthielen 9y ago
  Well I have been interested in programming languages for a very long time. I was hired into a group at Morgan Stanley specifically to develop this project. We had some initial use-cases where some internal trading systems could be "hot patched" with new logic but it was very slow (there was basically a simple AST-walking interpreter here). So at first I just had to make a roughly equivalent thing that could be much faster. However once that immediate need was satisfied, we found many more interesting places to apply this tool. I would say that the storage and live data-analysis features are just as valuable as the efficient machine code produced by hobbes. Yes I have used kdb/q before. It's pretty popular in the financial domain. I really like what we can do with type systems. :D Getting this project open sourced was really accomplished by other folks who I'm not sure if I can name here (I will check with them) but I was very happy to help make it happen. There are several folks internally who have been working really hard to make it possible for the firm to open source the work that we do and hobbes is just one part of that effort. There are several folks in management positions in the firm who have made open source efforts like this a priority and have even gone out on a limb to make sure that this happens. Morgan Stanley is a really remarkable company in the technology/finance domain, in my experience.
obl 9y ago
any plan to move beyond "allocate linearly and never free anything" memory management ? semantics look nice and coherent but unfortunately without some kind (automatic or manual) of memory management it's not usable as a general purpose tool.
- continuations 9y ago
  Wait, so there's not even manual memory management? I can't even call free()? How would this work then?
- kthielen 9y ago
  I wouldn't summarize it that way. We obviously "free" memory or else our trading systems wouldn't work and we'd be in the news in a bad way. :) Allocation does happen out of a default region and applications that embed hobbes generally define "transaction boundaries" for their use of compiled expressions which may allocate. It's also possible to manually nest these regions and control them yourself, which we do in some applications (e.g. where we're loading whole days of trading data and indexing them in batches). Ultimately we might formalize that process with a region inference algorithm.
eagle2001 9y ago
The LtU blurb says "a variant of Haskell", but the github readme does not mention non-strict. Strict or not?
- kthielen 9y ago
  Strict
Volt 9y ago
There was a paper a while back from someone working with MS on a Q typechecker in Prolog. Was this you?
- kthielen 9y ago
  Nope, that wasn't me. :)
willtim 9y ago
Are you able to comment on your implementation of row types? You give the following type for a field selector: > :t .foo (a.foo::b) => (a) -> b These give constraints with arbitrary types embedded in them. Most examples in the literature using qualified types, suggest something like the following type: (r lacks foo) => { foo :: b | r } -> b Where r is of kind row. The lacks constraint here is a very simple constraint and it is clear that this function takes a record, the disadvantage is that predicate lists can get very large. The old Haskell TREX extension built into Hugs worked this way. It would be great to learn some background behind your choice. Thanks.
- kthielen 9y ago
  Yes, TREX and Mark Jones's work on qualified types was very influential to this project. I am a big fan of that work. I added a few details in this post below: https://news.ycombinator.com/item?id=14787429 https://news.ycombinator.com/item?id=14787429 I hope that answers your questions although I know that there's a lot more that could be said about this topic.

ah- 9y ago

What motivated you to start building your own language? Did you use much kdb/q before? How did you manage to get this open sourced?

throwaway7645 9y ago
I'd like to know this as well. I know most array languages coupled with a DB aren't as fast as a language with a JIT, but the code is quite expressive. If Kdb+ can ever easily be run on an FPGA or GPU cluster...that would be something.
- stuntprogrammer 9y ago
  Not kdb+, but a proprietary (internal-only) language that it heavily influenced was designed around execution on GPU clusters. The FPGAs were used mainly for feedhandlers and there was a different DSL for that (compiling to verilog). It was indeed rather something to see :)
  - throwaway7645 9y ago
    Was it received favorably? A secret sauce, or a failed project.
    - stuntprogrammer 9y ago
      Secret sauce, at least for the team involved, and well-received. Nice combination of extreme perf and decent productivity for them. I've moved into non-finance stuff for quite a while now though, so not sure what's become of it. Given business challenges in that particular sub-field, who knows..
kthielen 9y ago
Well I have been interested in programming languages for a very long time. I was hired into a group at Morgan Stanley specifically to develop this project. We had some initial use-cases where some internal trading systems could be "hot patched" with new logic but it was very slow (there was basically a simple AST-walking interpreter here). So at first I just had to make a roughly equivalent thing that could be much faster. However once that immediate need was satisfied, we found many more interesting places to apply this tool. I would say that the storage and live data-analysis features are just as valuable as the efficient machine code produced by hobbes. Yes I have used kdb/q before. It's pretty popular in the financial domain. I really like what we can do with type systems. :D Getting this project open sourced was really accomplished by other folks who I'm not sure if I can name here (I will check with them) but I was very happy to help make it happen. There are several folks internally who have been working really hard to make it possible for the firm to open source the work that we do and hobbes is just one part of that effort. There are several folks in management positions in the firm who have made open source efforts like this a priority and have even gone out on a limb to make sure that this happens. Morgan Stanley is a really remarkable company in the technology/finance domain, in my experience.

obl 9y ago

any plan to move beyond "allocate linearly and never free anything" memory management ? semantics look nice and coherent but unfortunately without some kind (automatic or manual) of memory management it's not usable as a general purpose tool.

continuations 9y ago
Wait, so there's not even manual memory management? I can't even call free()? How would this work then?
kthielen 9y ago
I wouldn't summarize it that way. We obviously "free" memory or else our trading systems wouldn't work and we'd be in the news in a bad way. :) Allocation does happen out of a default region and applications that embed hobbes generally define "transaction boundaries" for their use of compiled expressions which may allocate. It's also possible to manually nest these regions and control them yourself, which we do in some applications (e.g. where we're loading whole days of trading data and indexing them in batches). Ultimately we might formalize that process with a region inference algorithm.

eagle2001 9y ago

The LtU blurb says "a variant of Haskell", but the github readme does not mention non-strict. Strict or not?

kthielen 9y ago
Strict

Volt 9y ago

There was a paper a while back from someone working with MS on a Q typechecker in Prolog. Was this you?

kthielen 9y ago
Nope, that wasn't me. :)

willtim 9y ago

Are you able to comment on your implementation of row types? You give the following type for a field selector: > :t .foo (a.foo::b) => (a) -> b These give constraints with arbitrary types embedded in them. Most examples in the literature using qualified types, suggest something like the following type: (r lacks foo) => { foo :: b | r } -> b Where r is of kind row. The lacks constraint here is a very simple constraint and it is clear that this function takes a record, the disadvantage is that predicate lists can get very large. The old Haskell TREX extension built into Hugs worked this way. It would be great to learn some background behind your choice. Thanks.

kthielen 9y ago
Yes, TREX and Mark Jones's work on qualified types was very influential to this project. I am a big fan of that work. I added a few details in this post below: https://news.ycombinator.com/item?id=14787429 https://news.ycombinator.com/item?id=14787429 I hope that answers your questions although I know that there's a lot more that could be said about this topic.