5 ms·

I’m happy someone mentioned the community in Julia! We’re always looking for interested people to join and try out some of the systems. For easy access to some

by mccoyb 6y ago

I’m happy someone mentioned the community in Julia! We’re always looking for interested people to join and try out some of the systems. For easy access to some of the active PP frameworks: Turing.jl https://turing.ml/dev/ https://turing.ml/dev/ Gen.jl https://www.gen.dev/ https://www.gen.dev/ Soss.jl https://github.com/cscherrer/Soss.jl https://github.com/cscherrer/Soss.jl Jaynes.jl https://github.com/femtomc/Jaynes.jl https://github.com/femtomc/Jaynes.jl Why so many? The design space is just beginning to be well-explored! And the community has welcomed experimentation! Gen.jl and Jaynes.jl are research projects aiming to push the boundary of what you can express inside a PP framework - inference here is sample-based, with incremental computation providing most of the optimization. Gen.jl is much more mature than the latter (which I develop) and has tons of great and thought-provoking documentation. Turing.jl is also very mature, and they have a great number of resources showing how to express a number of classic models in their language, as well as how to compose sampling-based inference algorithms. The Turing folks also implemented the de-facto version of HMC in Julia https://github.com/TuringLang/AdvancedHMC.jl https://github.com/TuringLang/AdvancedHMC.jl. Soss.jl is also very cool - and works by AST re-writing - with cool functionality integration from SymPy in Python. I’m not sure the state of the docs for Soss.jl - but the researchers for each of these systems are always willing to discuss the systems on the Julia Slack or Zulip!

cf 6y ago
The reason Julia is a good fit is it has both good numerics support (arrays, matrices, autodiff, samplers, etc) and good metaprogramming support. Probabilistic programming systems really need both to feel natural and well-integrated while still being able to have nice inference algorithms. A PPS in an environment without those two things is at a disadvantage.
- nightski 6y ago
  I am pretty new to all of this but I get the impression that Stan and PyMC3 are the leaders in this area and I don't see them as having great meta-programming support. Maybe I am wrong? Are they currently hitting limitations in that regard? Or is this in a particular area of PPL such as non-parametrics?
  - FridgeSeal 6y ago
    I don't know quite enough about the field, but it's possible they're the leaders (like Python) because they're just "good enough" and they were the best options widely available when they were adopted + took off?
    - cf 6y ago
      The reality is that most of a modelling task is preprocessing your data before it can be passed to a probabilistic model and postprocessing to make decisions using it. The code is usually written in R or Python so there is a strong pressure for your library to be in that language as well. And being rough around the edges is an ok price to pay for not losing an ecosystem.
      - bacr 6y ago
        My take is that most modern PPLs have language bindings in JavaScript/Python/R because they are explicitly courting analysts/data-scientists/applied-statisticians, or they are taking advantage of modern technical stacks implementing auto-diff and co-processor routines. Most pre- and post- processing (with the exception of visualization) should probably be a part of your model!
    - martintrapp 6y ago
      One of the arguments for python is the exceptional support of automation differentiation and GPU computing through deep learning libraries. Most python based PPLs focus on static model with differentiable log joints, allowing the application of HMC or variational inference. Unfortunately, the support of efficient automatic differentiation libraries in Julia is still in its infancy. But I hope with some more work by the community and the Turing team, this will change sooner than later.
      - cf 6y ago
        I thought with libraries like Zygote there is some really nice stuff already in Julia. I'd say it's still early days for good autodiff libraries in general and I think we still haven't really explored what they can do.
shakow 6y ago
It's probably just me being dense, but I don't understand how much prob. prog. differs from “traditional” programming. I just read the tutorials & examples for Turing and Gen, and to me, it seems that they are more or less nifty DSLs for expressing statistical models, but their functionalities could be easily replicated, albeit in an uglier way, with standard Julia. Is there some deeper theoretical divergence I missed, or are they just strongly facilitating the expression of statistical models?
- mccoyb 6y ago
  This basically applies to all programming? Reconsider this thought in the context of automatic differentiation, where a similar argument can be made. I doubt, however, that people would contest the utility of automatic differentiation systems. I also disagree that their functionality could be easily replicated in standard Julia. What you see is the easiest way to provide this functionality in a model-agnostic way.
  - shakow 6y ago
    > This basically applies to all programming? That's true. But if you should describe the essence of probabilistic programming to someone used only to “classical” scientific computing, what would be the key point(s)? BTW, I'm sorry if I came off as dismissive, I'm only frustrated not to grasp why it's considered to be such a step forward. AD I get it, all the differentials are automagically imputed directly from the source code, which is something that practically could not be done otherwise. But what I get from Turing or Gen are just nifty DSLs.
    - martintrapp 6y ago
      AD does automatic differentiation, PLLs transform a generative model into some suitable form to perform automatic Bayesian inference, e.g. by using AD and black-box variational inference. Or said differently, in a PPL you specify the forward simulation of a generative process and the PPL helps to automatically invert this process using black-box algorithms and suitable transformations. Without a PPL, you would traditionally write your code for your model and would have to implement a suitable inference algorithm yourself. With a PPL you only specify the generative process and don't have to implement the inference side of things nor care about an implementation of your model that is suitable for inference.
martintrapp 6y ago
Thanks for mentioning Turing. In comparison to other PPLs in Julia, Turing is less a single probabilistic programming library and more a framework for probabilistic programming by providing a large collection of exchangeable libraries. AdvancedHMC, which you mentioned, is simply one of the many projects for probabilistic machine learning that we actively develop. AdvancedHMC specifically is meant as a research platform for HMC algorithms and implements state of the art algorithms for HMC based inference. Other Turing libraries focus more towards variational approximations of static models, Bayesian learning in neural networks, and universal probabilistic programming with support for dynamic models.

by mccoyb 6y ago

cf 6y ago

The reason Julia is a good fit is it has both good numerics support (arrays, matrices, autodiff, samplers, etc) and good metaprogramming support. Probabilistic programming systems really need both to feel natural and well-integrated while still being able to have nice inference algorithms. A PPS in an environment without those two things is at a disadvantage.

nightski 6y ago
I am pretty new to all of this but I get the impression that Stan and PyMC3 are the leaders in this area and I don't see them as having great meta-programming support. Maybe I am wrong? Are they currently hitting limitations in that regard? Or is this in a particular area of PPL such as non-parametrics?
- FridgeSeal 6y ago
  I don't know quite enough about the field, but it's possible they're the leaders (like Python) because they're just "good enough" and they were the best options widely available when they were adopted + took off?
  - cf 6y ago
    The reality is that most of a modelling task is preprocessing your data before it can be passed to a probabilistic model and postprocessing to make decisions using it. The code is usually written in R or Python so there is a strong pressure for your library to be in that language as well. And being rough around the edges is an ok price to pay for not losing an ecosystem.
    - bacr 6y ago
      My take is that most modern PPLs have language bindings in JavaScript/Python/R because they are explicitly courting analysts/data-scientists/applied-statisticians, or they are taking advantage of modern technical stacks implementing auto-diff and co-processor routines. Most pre- and post- processing (with the exception of visualization) should probably be a part of your model!
  - martintrapp 6y ago
    One of the arguments for python is the exceptional support of automation differentiation and GPU computing through deep learning libraries. Most python based PPLs focus on static model with differentiable log joints, allowing the application of HMC or variational inference. Unfortunately, the support of efficient automatic differentiation libraries in Julia is still in its infancy. But I hope with some more work by the community and the Turing team, this will change sooner than later.
    - cf 6y ago
      I thought with libraries like Zygote there is some really nice stuff already in Julia. I'd say it's still early days for good autodiff libraries in general and I think we still haven't really explored what they can do.

shakow 6y ago

It's probably just me being dense, but I don't understand how much prob. prog. differs from “traditional” programming. I just read the tutorials & examples for Turing and Gen, and to me, it seems that they are more or less nifty DSLs for expressing statistical models, but their functionalities could be easily replicated, albeit in an uglier way, with standard Julia. Is there some deeper theoretical divergence I missed, or are they just strongly facilitating the expression of statistical models?

mccoyb 6y ago
This basically applies to all programming? Reconsider this thought in the context of automatic differentiation, where a similar argument can be made. I doubt, however, that people would contest the utility of automatic differentiation systems. I also disagree that their functionality could be easily replicated in standard Julia. What you see is the easiest way to provide this functionality in a model-agnostic way.
- shakow 6y ago
  > This basically applies to all programming? That's true. But if you should describe the essence of probabilistic programming to someone used only to “classical” scientific computing, what would be the key point(s)? BTW, I'm sorry if I came off as dismissive, I'm only frustrated not to grasp why it's considered to be such a step forward. AD I get it, all the differentials are automagically imputed directly from the source code, which is something that practically could not be done otherwise. But what I get from Turing or Gen are just nifty DSLs.
  - martintrapp 6y ago
    AD does automatic differentiation, PLLs transform a generative model into some suitable form to perform automatic Bayesian inference, e.g. by using AD and black-box variational inference. Or said differently, in a PPL you specify the forward simulation of a generative process and the PPL helps to automatically invert this process using black-box algorithms and suitable transformations. Without a PPL, you would traditionally write your code for your model and would have to implement a suitable inference algorithm yourself. With a PPL you only specify the generative process and don't have to implement the inference side of things nor care about an implementation of your model that is suitable for inference.

martintrapp 6y ago

Thanks for mentioning Turing. In comparison to other PPLs in Julia, Turing is less a single probabilistic programming library and more a framework for probabilistic programming by providing a large collection of exchangeable libraries. AdvancedHMC, which you mentioned, is simply one of the many projects for probabilistic machine learning that we actively develop. AdvancedHMC specifically is meant as a research platform for HMC algorithms and implements state of the art algorithms for HMC based inference. Other Turing libraries focus more towards variational approximations of static models, Bayesian learning in neural networks, and universal probabilistic programming with support for dynamic models.