3 ms·

This looks like the technology behind Prior Knowledge (http://priorknowledge.com/ http://priorknowledge.com/), which is not surprising given that a lot of the s

This looks like the technology behind Prior Knowledge (http://priorknowledge.com/ http://priorknowledge.com/), which is not surprising given that a lot of the same people are involved. Awesome that it's going open source. It will be interesting to see how far the database analogy can be pushed. The key thing to realize is that BayesDB is based on a particular model, CrossCat (http://probcomp.csail.mit.edu/crosscat/ http://probcomp.csail.mit.edu/crosscat/). If your database table is an adjacency list that represents a graph, for instance, it's not going to perform very well compared to a more tailored model. On the other hand, a generic approach to high-dimensional imputation is very useful.

stochastician 13y ago
Eric Jonas here, former CEO of Prior Knowledge, cofounder of Navia, etc. etc. [and obviously not speaking for my employer] Tristan is right, our tech was based on CrossCat, a model originally developed in Josh Tenenbaum's lab at MIT. We fundamentally believe that structured probabilistic models are the future -- this is very much the direction that my latest research has taken, for example. The reality is that almost any model that lets you build up a good joint density estimate of the data can be useful for this sort of tabular data. CrossCat assumes that your joint factors in to several independent factors -- that is, p(x, y, z, w) = p(x, y) p(z, w), for example. Of course, because it's a dirichlet process of dirichlet processes, all that's learned from the data. But that's not always the best approach, there are others, and it will be fun to see what else comes out of the research community. And Tristan's also right that this is not always a great fit for your data -- if it's graph based, or if your data are super-high-dimensional, or whatever, it's not a perfect fit. But it is a good start. For a great review of the Dirichlet Process/CRP, the Indian Buffet Process, and other probability distributions over infinite structures, I highly recommend this great tech report from Zoubin and Tom: http://mlg.eng.cam.ac.uk/pub/pdf/GriGha11.pdf http://mlg.eng.cam.ac.uk/pub/pdf/GriGha11.pdf
- jaybaxter 13y ago
  Agreed! CrossCat works really well for many datasets, but I am also very excited about implementing BayesDB with other joint density estimators.

stochastician 13y ago

Eric Jonas here, former CEO of Prior Knowledge, cofounder of Navia, etc. etc. [and obviously not speaking for my employer] Tristan is right, our tech was based on CrossCat, a model originally developed in Josh Tenenbaum's lab at MIT. We fundamentally believe that structured probabilistic models are the future -- this is very much the direction that my latest research has taken, for example. The reality is that almost any model that lets you build up a good joint density estimate of the data can be useful for this sort of tabular data. CrossCat assumes that your joint factors in to several independent factors -- that is, p(x, y, z, w) = p(x, y) p(z, w), for example. Of course, because it's a dirichlet process of dirichlet processes, all that's learned from the data. But that's not always the best approach, there are others, and it will be fun to see what else comes out of the research community. And Tristan's also right that this is not always a great fit for your data -- if it's graph based, or if your data are super-high-dimensional, or whatever, it's not a perfect fit. But it is a good start. For a great review of the Dirichlet Process/CRP, the Indian Buffet Process, and other probability distributions over infinite structures, I highly recommend this great tech report from Zoubin and Tom: http://mlg.eng.cam.ac.uk/pub/pdf/GriGha11.pdf http://mlg.eng.cam.ac.uk/pub/pdf/GriGha11.pdf

jaybaxter 13y ago
Agreed! CrossCat works really well for many datasets, but I am also very excited about implementing BayesDB with other joint density estimators.