6 ms·

BayesDB: A probabilistic programming platform

222 points by erwan 9y ago

3pt14159 9y ago
Welcome to the project I've been waiting for years to get out of alpha. It's frustrating. If I had a hundred million dollars I'd burn a couple million getting this funded. It seems like it will be useful to humanity.
- carterschonwald 9y ago
  ... they are funded. (I know of >=1 of the investors) https://empirical.com/ https://empirical.com/
mayneack 9y ago
previous discussions: https://news.ycombinator.com/item?id=6864339 https://news.ycombinator.com/item?id=6864339 https://news.ycombinator.com/item?id=10750900 https://news.ycombinator.com/item?id=10750900
indescions_2017 9y ago
Glad to see this is out as well! Using probabilistic frameworks has the potential to eliminate a lot of the human error which can easily enter a large simulation. It's fair to say in the future probabilistic modules will become part of every standard library in every programming language, and distribution sampling functions will be as common as trig functions in a math library. I am curious though how I would build up large queries in the BQL (SQL-like query language) or MML (meta-modeling language). For the orbital example, we conceivably only have a relatively low dimensional space. But what about a Bayes net for investigating genetic variants in a large genomic population? Doesn't this quickly become intractable?
- mozartoz 9y ago
  Some probabilisitic systems can cope with big volumes of data. E.g. http://i.stanford.edu/hazy/ http://i.stanford.edu/hazy/
  - sanxiyn 9y ago
    Note that Apple recently acquired this technology. Lattice Data is the commercialization of DeepDive.
afpx 9y ago
Anyone doing probabilistic programming with big data? I started experimenting with probabilistic programming frameworks several years ago, but couldn't get it to scale to the level of data I'm working with (~6 dimensions ~100 trillion vectors), or even a small fraction of that. But, I'm sure it's being done in scientific circles somewhere. Are there communities to collaborate on probabilistic programming? It seems like he domain knowledge is obscure enough that all the good information is locked up in the big corporations and academics.
- currymj 9y ago
  Things have gotten much more mature in the past few years. Check out edwardlib.org which is adapting Tensorflow to support probabilistic modeling (with a heavy focus on variational inference, less so on MCMC methods). If you’ve got trillions of observations you can use stochastic VI. And tensorflow now can do distributed computation graphs, or you could just go data parallel and then average your parameters at the end. In general David Blei’s group at Columbia does a lot of work in scalable probabilistic inference. The other big option is of course Stan, which is really well optimized but I don’t think is particularly intended for “big data” of this kind. If you have “medium data” that fits on one machine though, it’s blazing fast.
  - 2309kdujj 9y ago
    Funny you mention those. I've been really excited about Edward but when I tried it for a project last year I could never get it to come together in the right way. I got the sense that it wasn't quite ready for prime time yet, although very promising. My memory of it was that a lot of claimed flexibility in how to specify models wasn't really implemented fully. The experience also turned me off of TensorFlow a bit. But that was a year ago, so maybe it's improved? I ended up doing it in Stan in part because I was more familiar with that, and it worked out fairly well. Just a personal anecdote.
- nightski 9y ago
  I have not, so take my words with a grain of salt. However, with only 6 dimensions 100 trillion vectors is probably extremely overkill (unless you are talking time series data?). Probabilistic programming, from what I understand, thrives on situations where you do not have a lot of data (instead of an overabundance). You could probably achieve very interesting results by taking a much smaller sample from your large data set.
elvinyung 9y ago
They really missed out on an opportunity to call it DataBayes.
stanfordkid 9y ago
This sounds kind of similar to the stuff this startup called "Prior Knowledge" was working on prior to being acquired by Salesforce: https://www.crunchbase.com/organization/prior-knowledge#/entity https://www.crunchbase.com/organization/prior-knowledge#/ent...
kensai 9y ago
Is there a comparison of its accuracy against traditional methods? Admittedly, this machine assisted modeling sounds really interesting.
jarym 9y ago
I can imagine this could be really great wrapped up as a Postgres extension

222 points by erwan 9y ago

Welcome to the project I've been waiting for years to get out of alpha. It's frustrating. If I had a hundred million dollars I'd burn a couple million getting this funded. It seems like it will be useful to humanity.

carterschonwald 9y ago
... they are funded. (I know of >=1 of the investors) https://empirical.com/ https://empirical.com/

mayneack 9y ago

previous discussions: https://news.ycombinator.com/item?id=6864339 https://news.ycombinator.com/item?id=6864339 https://news.ycombinator.com/item?id=10750900 https://news.ycombinator.com/item?id=10750900

indescions_2017 9y ago

Glad to see this is out as well! Using probabilistic frameworks has the potential to eliminate a lot of the human error which can easily enter a large simulation. It's fair to say in the future probabilistic modules will become part of every standard library in every programming language, and distribution sampling functions will be as common as trig functions in a math library. I am curious though how I would build up large queries in the BQL (SQL-like query language) or MML (meta-modeling language). For the orbital example, we conceivably only have a relatively low dimensional space. But what about a Bayes net for investigating genetic variants in a large genomic population? Doesn't this quickly become intractable?

mozartoz 9y ago
Some probabilisitic systems can cope with big volumes of data. E.g. http://i.stanford.edu/hazy/ http://i.stanford.edu/hazy/
- sanxiyn 9y ago
  Note that Apple recently acquired this technology. Lattice Data is the commercialization of DeepDive.

afpx 9y ago

Anyone doing probabilistic programming with big data? I started experimenting with probabilistic programming frameworks several years ago, but couldn't get it to scale to the level of data I'm working with (~6 dimensions ~100 trillion vectors), or even a small fraction of that. But, I'm sure it's being done in scientific circles somewhere. Are there communities to collaborate on probabilistic programming? It seems like he domain knowledge is obscure enough that all the good information is locked up in the big corporations and academics.

currymj 9y ago
Things have gotten much more mature in the past few years. Check out edwardlib.org which is adapting Tensorflow to support probabilistic modeling (with a heavy focus on variational inference, less so on MCMC methods). If you’ve got trillions of observations you can use stochastic VI. And tensorflow now can do distributed computation graphs, or you could just go data parallel and then average your parameters at the end. In general David Blei’s group at Columbia does a lot of work in scalable probabilistic inference. The other big option is of course Stan, which is really well optimized but I don’t think is particularly intended for “big data” of this kind. If you have “medium data” that fits on one machine though, it’s blazing fast.
- 2309kdujj 9y ago
  Funny you mention those. I've been really excited about Edward but when I tried it for a project last year I could never get it to come together in the right way. I got the sense that it wasn't quite ready for prime time yet, although very promising. My memory of it was that a lot of claimed flexibility in how to specify models wasn't really implemented fully. The experience also turned me off of TensorFlow a bit. But that was a year ago, so maybe it's improved? I ended up doing it in Stan in part because I was more familiar with that, and it worked out fairly well. Just a personal anecdote.
nightski 9y ago
I have not, so take my words with a grain of salt. However, with only 6 dimensions 100 trillion vectors is probably extremely overkill (unless you are talking time series data?). Probabilistic programming, from what I understand, thrives on situations where you do not have a lot of data (instead of an overabundance). You could probably achieve very interesting results by taking a much smaller sample from your large data set.

elvinyung 9y ago

They really missed out on an opportunity to call it DataBayes.

stanfordkid 9y ago

This sounds kind of similar to the stuff this startup called "Prior Knowledge" was working on prior to being acquired by Salesforce: https://www.crunchbase.com/organization/prior-knowledge#/entity https://www.crunchbase.com/organization/prior-knowledge#/ent...

kensai 9y ago

Is there a comparison of its accuracy against traditional methods? Admittedly, this machine assisted modeling sounds really interesting.

jarym 9y ago

I can imagine this could be really great wrapped up as a Postgres extension