5 ms·

The Algorithms Behind Probabilistic Programming

329 points by williamsmj 10y ago

joe_the_user 10y ago
Nice, The algorithms the article define seem to justify seeing probabilistic programming as an advance beyond twenty line programs that call 20K sized libraries - basically, these algorithm inherently auto-tune a lot of higher level operations so these can be less-leaky abstractions.
misiti3780 10y ago
nice article - please change your font - it makes reading those blog post very difficult
- aswanson 10y ago
  Hilarious, I was just going to comment on how cool/80s retro I thought the fonts were.
- ntumlin 10y ago
  What did you dislike about the font? I had no trouble reading it.
  - sp332 10y ago
    It has a lot of straight lines where my eyes expect curves. It really slowed me down reading, so I switched to readability mode.
- hmottestad 10y ago
  I also did not like the font, hard to say why. It definitely is not designed with being legible in mind. Feels very unbalanced. I think the biggest issue is with the distracting curves, and varying ways of drawing the various curves. Kerning is also messed up, the distance between the letters is too large in comparison to the distance between the words.
  - Thirdegree 10y ago
    It feel like a normal font that squished each character into a slightly-too-small box.
- pliftkl 10y ago
  Glad I'm not the only one. It's not often that I run into fonts that slow me down, but that hurt to read.
adamnemecek 10y ago
You should check out http://webppl.org/ http://webppl.org/, you implement your own probabilistic programming language in javascript.
sn6uv 10y ago
Nit > The distribution must be smooth (this method doesn’t work for discrete parameters), but it need not be analytically differentiable 'Smooth' and 'analytic' both have basic technical meanings in the context of differentiability [1], [2]. Glossing over the assumptions in this way is quite misleading [1] https://en.m.wikipedia.org/wiki/Smoothness https://en.m.wikipedia.org/wiki/Smoothness [2] https://en.m.wikipedia.org/wiki/Analytic_function https://en.m.wikipedia.org/wiki/Analytic_function
- mr_pink 10y ago
  Analytically differentiable (being able to derive a formula for the derivative) is not the same thing at all as analytic functions of complex analysis (which are infinitely differentiable, etc). If anything, the term "analytic function" seems more of a misnomer to me, but "analytically differentiable" is used in its proper sense in the article.
riazrizvi 10y ago
Interesting marketing. It seems to be an API for statistical functions and data. Though the term Probabilistic Programming sugggests a new programming paradigm, which it is not. If I'm not missing something, the term makes a dry subject sound cool.
- williamsmj 10y ago
  There's much more to it than you suggest, but if marketing bothers you then check out the MIT Probabilistic Programming Project's talk at Strange Loop 2015: https://www.youtube.com/watch?v=-8QMqSWU76Q https://www.youtube.com/watch?v=-8QMqSWU76Q. Probabilistic programming is an area of huge academic activity, with deep connections to statistical inference, compiler design and, yes, UX.
  - riazrizvi 10y ago
    Oh I see. It is now a recognized application field in CS like Big Data, due to maturity of a suite of different tools from APIs to data engines.
  - cscurmudgeon 10y ago
    Academia is not exempt from marketing. And not all marketing is bad. :) Remember the origin of the term "dynamic programming"? > "I spent the Fall quarter (of 1950) at RAND. My first task was to find a name for multistage decision processes. An interesting question is, Where did the name, dynamic programming, come from? The 1950s were not good years for mathematical research. We had a very interesting gentleman in Washington named Wilson. He was Secretary of Defense, and he actually had a pathological fear and hatred of the word research. I’m not using the term lightly; I’m using it precisely. His face would suffuse, he would turn red, and he would get violent if people used the term research in his presence. You can imagine how he felt, then, about the term mathematical. The RAND Corporation was employed by the Air Force, and the Air Force had Wilson as its boss, essentially. Hence, I felt I had to do something to shield Wilson and the Air Force from the fact that I was really doing mathematics inside the RAND Corporation. What title, what name, could I choose? In the first place I was interested in planning, in decision making, in thinking. But planning, is not a good word for various reasons. I decided therefore to use the word “programming”. I wanted to get across the idea that this was dynamic, this was multistage, this was time-varying. I thought, let's kill two birds with one stone. Let's take a word that has an absolutely precise meaning, namely dynamic, in the classical physical sense. It also has a very interesting property as an adjective, and that it's impossible to use the word dynamic in a pejorative sense. Try thinking of some combination that will possibly give it a pejorative meaning. It's impossible. Thus, I thought dynamic programming was a good name. It was something not even a Congressman could object to. So I used it as an umbrella for my activities." https://en.wikipedia.org/wiki/Dynamic_programming#History https://en.wikipedia.org/wiki/Dynamic_programming#History
davidw 10y ago
Why is this flagged?! Cursory glance at the article looked like perfect HN material.
- laxatives 10y ago
  The article consists of a list of algorithms with almost no other information. Its posted because it has a link to the OP's own blog.
  - platz 10y ago
    It was helpful to me, it put these algorithms in context and explained why there are useful over other methods. Were you expecting a graduate level course embedded in the post?
3pt14159 10y ago
It's taking a long time to come out, but the probabilistic approach I think is the most interesting is BayesDB: http://probcomp.csail.mit.edu/bayesdb/ http://probcomp.csail.mit.edu/bayesdb/ Almost always what I want is closer to SQL-like questions that I later want to manipulate with languages like Ruby or Python, rather than having to load all the data into a giant array and then have to use probabilistic programming on the whole damn thing.

329 points by williamsmj 10y ago

Nice, The algorithms the article define seem to justify seeing probabilistic programming as an advance beyond twenty line programs that call 20K sized libraries - basically, these algorithm inherently auto-tune a lot of higher level operations so these can be less-leaky abstractions.

misiti3780 10y ago

nice article - please change your font - it makes reading those blog post very difficult

aswanson 10y ago
Hilarious, I was just going to comment on how cool/80s retro I thought the fonts were.
ntumlin 10y ago
What did you dislike about the font? I had no trouble reading it.
- sp332 10y ago
  It has a lot of straight lines where my eyes expect curves. It really slowed me down reading, so I switched to readability mode.
hmottestad 10y ago
I also did not like the font, hard to say why. It definitely is not designed with being legible in mind. Feels very unbalanced. I think the biggest issue is with the distracting curves, and varying ways of drawing the various curves. Kerning is also messed up, the distance between the letters is too large in comparison to the distance between the words.
- Thirdegree 10y ago
  It feel like a normal font that squished each character into a slightly-too-small box.
pliftkl 10y ago
Glad I'm not the only one. It's not often that I run into fonts that slow me down, but that hurt to read.

adamnemecek 10y ago

You should check out http://webppl.org/ http://webppl.org/, you implement your own probabilistic programming language in javascript.

sn6uv 10y ago

Nit > The distribution must be smooth (this method doesn’t work for discrete parameters), but it need not be analytically differentiable 'Smooth' and 'analytic' both have basic technical meanings in the context of differentiability [1], [2]. Glossing over the assumptions in this way is quite misleading [1] https://en.m.wikipedia.org/wiki/Smoothness https://en.m.wikipedia.org/wiki/Smoothness [2] https://en.m.wikipedia.org/wiki/Analytic_function https://en.m.wikipedia.org/wiki/Analytic_function

mr_pink 10y ago
Analytically differentiable (being able to derive a formula for the derivative) is not the same thing at all as analytic functions of complex analysis (which are infinitely differentiable, etc). If anything, the term "analytic function" seems more of a misnomer to me, but "analytically differentiable" is used in its proper sense in the article.

riazrizvi 10y ago

Interesting marketing. It seems to be an API for statistical functions and data. Though the term Probabilistic Programming sugggests a new programming paradigm, which it is not. If I'm not missing something, the term makes a dry subject sound cool.

williamsmj 10y ago
There's much more to it than you suggest, but if marketing bothers you then check out the MIT Probabilistic Programming Project's talk at Strange Loop 2015: https://www.youtube.com/watch?v=-8QMqSWU76Q https://www.youtube.com/watch?v=-8QMqSWU76Q. Probabilistic programming is an area of huge academic activity, with deep connections to statistical inference, compiler design and, yes, UX.
- riazrizvi 10y ago
  Oh I see. It is now a recognized application field in CS like Big Data, due to maturity of a suite of different tools from APIs to data engines.
- cscurmudgeon 10y ago
  Academia is not exempt from marketing. And not all marketing is bad. :) Remember the origin of the term "dynamic programming"? > "I spent the Fall quarter (of 1950) at RAND. My first task was to find a name for multistage decision processes. An interesting question is, Where did the name, dynamic programming, come from? The 1950s were not good years for mathematical research. We had a very interesting gentleman in Washington named Wilson. He was Secretary of Defense, and he actually had a pathological fear and hatred of the word research. I’m not using the term lightly; I’m using it precisely. His face would suffuse, he would turn red, and he would get violent if people used the term research in his presence. You can imagine how he felt, then, about the term mathematical. The RAND Corporation was employed by the Air Force, and the Air Force had Wilson as its boss, essentially. Hence, I felt I had to do something to shield Wilson and the Air Force from the fact that I was really doing mathematics inside the RAND Corporation. What title, what name, could I choose? In the first place I was interested in planning, in decision making, in thinking. But planning, is not a good word for various reasons. I decided therefore to use the word “programming”. I wanted to get across the idea that this was dynamic, this was multistage, this was time-varying. I thought, let's kill two birds with one stone. Let's take a word that has an absolutely precise meaning, namely dynamic, in the classical physical sense. It also has a very interesting property as an adjective, and that it's impossible to use the word dynamic in a pejorative sense. Try thinking of some combination that will possibly give it a pejorative meaning. It's impossible. Thus, I thought dynamic programming was a good name. It was something not even a Congressman could object to. So I used it as an umbrella for my activities." https://en.wikipedia.org/wiki/Dynamic_programming#History https://en.wikipedia.org/wiki/Dynamic_programming#History

davidw 10y ago

Why is this flagged?! Cursory glance at the article looked like perfect HN material.

laxatives 10y ago
The article consists of a list of algorithms with almost no other information. Its posted because it has a link to the OP's own blog.
- platz 10y ago
  It was helpful to me, it put these algorithms in context and explained why there are useful over other methods. Were you expecting a graduate level course embedded in the post?

3pt14159 10y ago

It's taking a long time to come out, but the probabilistic approach I think is the most interesting is BayesDB: http://probcomp.csail.mit.edu/bayesdb/ http://probcomp.csail.mit.edu/bayesdb/ Almost always what I want is closer to SQL-like questions that I later want to manipulate with languages like Ruby or Python, rather than having to load all the data into a giant array and then have to use probabilistic programming on the whole damn thing.