3 ms·

The Gibbs sampler is right at the top of my personal list of the greatest technical achievements in statistics. Not only is the asymptotic guarantee of producin

The Gibbs sampler is right at the top of my personal list of the greatest technical achievements in statistics. Not only is the asymptotic guarantee of producing a sample from the target distribution a neat and elegant result, but it also opened up a really rich world of probabilistic modeling. From a practical / ML perspective, variational methods are more efficient for getting decent estimates of complex models. And there are a lot of devilish details to MCMC sampling, and the real world _never_ goes to infinity. Still, the sheer theoretical fact that an imperfect, simulation-based method can produce valid statistical estimates is just awesome.

wheresmycraisin 6y ago
I appreciate the theoretical aspects of it as well -- however, I could never get over the notion that bayesians need to use frequentist statistics to judge whether the chain has sufficiently converged... isn't that defeating the entire point?
- soVeryTired 6y ago
  Interesting point, but I think the academic culture has taken a pragmatic shift: there aren't many 'pure' bayesians or frequentists left anymore, and the boundary between the two viewpoints has softened. Stats has more of a 'shut up and calculate' mentality about it these days.
  - sgt101 6y ago
    >Stats has more of a 'shut up and calculate' mentality about it these days. I'm happy that the "pure this pure that" fundamentalism has faded into retirement... but shut up and calculate has problems - especially around causality. The "blessings of multiple causes" paper for example.
  - wheresmycraisin 6y ago
    Then why bother pretending to calculate a posterior distribution when you can do a MLE estimator with regularization or something?
frodetb 6y ago
I wrote my masters on using simulated annealing to solve a particular optimisation problem. In a sense, the Gibbs sampler lies at its core, and I really have to agree with you in what you say. At the time I did not have time to think too deeply or too much about it, but I have since really been increasingly awestruck with it, with thermodynamics and statistical mechanics in general, and analogies that can be drawn to all kinds of disciplines.
- tetris11 6y ago
  In bioinformatics, MCMC was used often linkage analysis to sample a "genotype space" of a true inheritance pattern of multiple related individuals to find the location and transmission pathway of disease causing variants. Classical genetics and statistics were very tightly linked in this regard due to scarcity of data at the time (pre-2000). Now that the field has exploded with big data, such statistics are no longer so neccesary to answer the same problems, and a lot of the early mathematical elegance associated with the field has given way to brute-force filtering methods.

wheresmycraisin 6y ago

I appreciate the theoretical aspects of it as well -- however, I could never get over the notion that bayesians need to use frequentist statistics to judge whether the chain has sufficiently converged... isn't that defeating the entire point?

soVeryTired 6y ago
Interesting point, but I think the academic culture has taken a pragmatic shift: there aren't many 'pure' bayesians or frequentists left anymore, and the boundary between the two viewpoints has softened. Stats has more of a 'shut up and calculate' mentality about it these days.
- sgt101 6y ago
  >Stats has more of a 'shut up and calculate' mentality about it these days. I'm happy that the "pure this pure that" fundamentalism has faded into retirement... but shut up and calculate has problems - especially around causality. The "blessings of multiple causes" paper for example.
- wheresmycraisin 6y ago
  Then why bother pretending to calculate a posterior distribution when you can do a MLE estimator with regularization or something?

frodetb 6y ago

I wrote my masters on using simulated annealing to solve a particular optimisation problem. In a sense, the Gibbs sampler lies at its core, and I really have to agree with you in what you say. At the time I did not have time to think too deeply or too much about it, but I have since really been increasingly awestruck with it, with thermodynamics and statistical mechanics in general, and analogies that can be drawn to all kinds of disciplines.

tetris11 6y ago
In bioinformatics, MCMC was used often linkage analysis to sample a "genotype space" of a true inheritance pattern of multiple related individuals to find the location and transmission pathway of disease causing variants. Classical genetics and statistics were very tightly linked in this regard due to scarcity of data at the time (pre-2000). Now that the field has exploded with big data, such statistics are no longer so neccesary to answer the same problems, and a lot of the early mathematical elegance associated with the field has given way to brute-force filtering methods.