4 ms·

They've really taken off in niche places. If you have a complex model of something, it's dramatically easier to use of of these to build/fit your model than it

They've really taken off in niche places. If you have a complex model of something, it's dramatically easier to use of of these to build/fit your model than it is to code by hand. But those cases are still things were you might have just a dozen variables (though each might be a long vector). It's more the realm of statistical inference than it is general programming or ML. It hasn't "taken off" in ML because ML problems generally have more specific solutions based on the problem. If you have something simple and tabular, other solutions are generally better. If you have something recsys shaped, other solutions are generally better. If you have something vision/language shaped, other solutions are generally better. It hasn't "taken off" in general programming because PPLs generally have trouble with control flow. Cutting off an entire arm of a program is trivial in a traditional language, but in PPLs you'll have to evaluate both. If the arm is a recursion step and hitting the base case is probabilistic, you might even have to evaluate arbitrarily deep (or you approximate that in a way that significantly limits the breadth of techniques available for running a program). AFAICT, a truism in PPL is that there are always programs that your language will run poorly on but a bespoke engine will do better, by an extreme margin. There just aren't general languages that perform as reliably as in deterministic languages. It's also just really really hard. It's roughly impossible to make things that are easy in normal languages easy to work with in PPLs. Consider these examples: `def f(x, y): return x + y + noise` where you condition on `f(3, y) == 5`. It's easy. `def f(password, salt): return hash(password + salt)` where you condition on `f(password, 8123746) == 1293487`. It's basically not going to happen even though forward evaluation of f is straightforward in any traditional language. Hell, even just supporting `def f(x, y): return x+y` is hard to generalize. Surprisingly it's harder to generalize than the `x+y+noise` case.

mccoyb 3y ago
I think you’re overgeneralizing in your control flow discussion. I also don’t understand your f example with (x, y, noise) if you fix x and the return value, you still have two unknowns with 1 equation. How is that easy to solve? Unless you’re considering using parametric inverses to represent the solution — but you didn’t mention this so I assume you didn’t mean this.
- 6gvONxR4sf7o 3y ago
  I was being underspecific to be concise, but in pyro, I'd mean something like this: def f(x, y): return pyro.sample( "z", dist.Normal(x+y, 1), obs=5 ) model = pyro.condition(f, x=3)

mccoyb 3y ago

I think you’re overgeneralizing in your control flow discussion. I also don’t understand your f example with (x, y, noise) if you fix x and the return value, you still have two unknowns with 1 equation. How is that easy to solve? Unless you’re considering using parametric inverses to represent the solution — but you didn’t mention this so I assume you didn’t mean this.

6gvONxR4sf7o 3y ago
I was being underspecific to be concise, but in pyro, I'd mean something like this: def f(x, y): return pyro.sample( "z", dist.Normal(x+y, 1), obs=5 ) model = pyro.condition(f, x=3)