3 ms·

The old calculation relies on older experimental results that have been verified by multiple experiments - so if the older value is wrong, it means either the c

by atty 6y ago

The old calculation relies on older experimental results that have been verified by multiple experiments - so if the older value is wrong, it means either the calculation was done wrong (possible), or the experiments all have had a significant correlated systematic error that has never been caught (also possible). However, I’d say both of those things are relatively unlikely, when compared to the probability of some small error in a new paper that was just released that uses a new method that involves lattice calculations. This is all a balance of probabilities argument, but from my experience in the field, I’d say it’s more likely that any errors in calculation or missed systematics would be in the new paper. However, I’m an experimentalist who has worked close to a lot of this stuff, not an actual theorist, so I’d love to get a theorists interpretation as well.

evanb 6y ago
I'm a lattice QCD practitioner. What I'll say is that the BMW collaboration isn't named that by coincidence---they're a resource-rich, extremely knowledgeable, cutting-edge group that is the envy of many others. They're also cut-throat competitive, which is very divisive. Grad students and postdocs are forced to sign NDAs to work on the hot stuff. That's insane. What's worse, from my point of view (as an actual LQCD practitioner) is: they're not very open about the actual details of their computation. It's tricky, because they treat their code as their 'secret sauce'. (Most of the community co-develops at least the base-level libraries; BMW goes it alone.) OK, so they don't want to share their source code; that's fine. But they ALSO don't want to share any of their gauge configurations (read: monte carlo samples) because they're expensive to produce and can be reused for other calculations. So it'd be frustrating to share your own resource-intensive products and have someone else scoop you with them. I disagree with that, but I get it at least. My biggest problem, and the one that I do not understand, is their reluctance to share the individual measurements they've made on each Monte Carlo sample. Then, at least, a motivated critic could develop their own statistical analysis (even if they can't develop their whole from-scratch computation). Because of the structure and workflow of a LQCD calculation it's very difficult to blind. So, the only thing I know to do is to say "here are all the inputs, at the bit-exact level, to our analysis, here are our analysis scripts, here's the result we get; see if you agree." This is the approach my collaborators and I took when we published a 1% determination of the nucleon axial coupling g_A [Nature 558, 91-94 (2018)]: we put the raw correlation functions as well as scripts on github https://github.com/callat-qcd/project_gA https://github.com/callat-qcd/project_gA and said "look, here's literally exactly what we do; if you run this you will get the numbers in the paper." It's not great because our analysis code isn't the cleanest thing in the world (we're more interested in results than in nice software engineering). But at least the raw data is right there, we tell you what each data set is, and you're free to analyze it. BMW does nothing of the sort. They (meaning those with power to dictate how the collaboration operates) seem to not want to adopt principles of nothing-up-my-sleeve really-honestly-truly open science. So their results need to be treated with care. That said, they themselves are extremely rigorous, top-notch scientists. They want you to trust them. Not that you shouldn't. Trust---but verify. That's currently not possible. I bet they're vindicated. But I can't check for myself.
- maxnoe 6y ago
  > OK, so they don't want to share their source code; that's fine. No, it is not. It is the exact reason why rheir results are not trust worthy. Publish the code, let it be checked by the peers. Closed source code has no place in science and most journals now rightly demand open code for the publications.
  - evanb 6y ago
    I appreciate the absolutist position. However, everybody agrees on what code _must do_. If you need to solve a (massive) system of linear equations (as happens often in LQCD) you can then take your alleged solution and plug it in and check. A variety of those sorts of things prevent you from doing anything too wrong. If you screw up gauge invariance, for example, you will get 0. There are agreed-upon small examples. Plus other benchmarks---they computed the hadronic spectrum. They computed the splitting between the proton's mass and the neutron's mass. If you spend hundreds or thousands of man hours optimizing, for example, assembler for a communications-intensive highly-parallel linear solve, it's fair to be reluctant to give it away. If you do others will get the glory (publications / funding). Some people do [ eg. this solver library for the BlueGenes https://www2.ph.ed.ac.uk/~paboyle/bagel/Bagel.html https://www2.ph.ed.ac.uk/~paboyle/bagel/Bagel.html ]. Most people are happy to let others do the hard work of building low-level libraries. But they COULD decide to write custom software that'd go faster. If their custom software reproduce results that community-standard libraries produce that's not nothing.
    - dguest 6y ago
      I think this clarifies a misunderstanding I had from your original comment. It sounds like the "secret sauce" for this collaboration includes a set of numerical libraries. They would get relatively little funding, few publications ("glory", as you say), and at best be reduced to a citation (if people remember to cite their libraries) if all they did was improve the backbone of lattice QCD with better software. So instead they keep it internal. It's a bit sad that there's so little glory in writing better numerical libraries, but it's a common problem across the sciences (and in the open source community in general) so I can believe they'd be reluctant to share.

by atty 6y ago

evanb 6y ago

I'm a lattice QCD practitioner. What I'll say is that the BMW collaboration isn't named that by coincidence---they're a resource-rich, extremely knowledgeable, cutting-edge group that is the envy of many others. They're also cut-throat competitive, which is very divisive. Grad students and postdocs are forced to sign NDAs to work on the hot stuff. That's insane. What's worse, from my point of view (as an actual LQCD practitioner) is: they're not very open about the actual details of their computation. It's tricky, because they treat their code as their 'secret sauce'. (Most of the community co-develops at least the base-level libraries; BMW goes it alone.) OK, so they don't want to share their source code; that's fine. But they ALSO don't want to share any of their gauge configurations (read: monte carlo samples) because they're expensive to produce and can be reused for other calculations. So it'd be frustrating to share your own resource-intensive products and have someone else scoop you with them. I disagree with that, but I get it at least. My biggest problem, and the one that I do not understand, is their reluctance to share the individual measurements they've made on each Monte Carlo sample. Then, at least, a motivated critic could develop their own statistical analysis (even if they can't develop their whole from-scratch computation). Because of the structure and workflow of a LQCD calculation it's very difficult to blind. So, the only thing I know to do is to say "here are all the inputs, at the bit-exact level, to our analysis, here are our analysis scripts, here's the result we get; see if you agree." This is the approach my collaborators and I took when we published a 1% determination of the nucleon axial coupling g_A [Nature 558, 91-94 (2018)]: we put the raw correlation functions as well as scripts on github https://github.com/callat-qcd/project_gA https://github.com/callat-qcd/project_gA and said "look, here's literally exactly what we do; if you run this you will get the numbers in the paper." It's not great because our analysis code isn't the cleanest thing in the world (we're more interested in results than in nice software engineering). But at least the raw data is right there, we tell you what each data set is, and you're free to analyze it. BMW does nothing of the sort. They (meaning those with power to dictate how the collaboration operates) seem to not want to adopt principles of nothing-up-my-sleeve really-honestly-truly open science. So their results need to be treated with care. That said, they themselves are extremely rigorous, top-notch scientists. They want you to trust them. Not that you shouldn't. Trust---but verify. That's currently not possible. I bet they're vindicated. But I can't check for myself.

maxnoe 6y ago
> OK, so they don't want to share their source code; that's fine. No, it is not. It is the exact reason why rheir results are not trust worthy. Publish the code, let it be checked by the peers. Closed source code has no place in science and most journals now rightly demand open code for the publications.
- evanb 6y ago
  I appreciate the absolutist position. However, everybody agrees on what code _must do_. If you need to solve a (massive) system of linear equations (as happens often in LQCD) you can then take your alleged solution and plug it in and check. A variety of those sorts of things prevent you from doing anything too wrong. If you screw up gauge invariance, for example, you will get 0. There are agreed-upon small examples. Plus other benchmarks---they computed the hadronic spectrum. They computed the splitting between the proton's mass and the neutron's mass. If you spend hundreds or thousands of man hours optimizing, for example, assembler for a communications-intensive highly-parallel linear solve, it's fair to be reluctant to give it away. If you do others will get the glory (publications / funding). Some people do [ eg. this solver library for the BlueGenes https://www2.ph.ed.ac.uk/~paboyle/bagel/Bagel.html https://www2.ph.ed.ac.uk/~paboyle/bagel/Bagel.html ]. Most people are happy to let others do the hard work of building low-level libraries. But they COULD decide to write custom software that'd go faster. If their custom software reproduce results that community-standard libraries produce that's not nothing.
  - dguest 6y ago
    I think this clarifies a misunderstanding I had from your original comment. It sounds like the "secret sauce" for this collaboration includes a set of numerical libraries. They would get relatively little funding, few publications ("glory", as you say), and at best be reduced to a citation (if people remember to cite their libraries) if all they did was improve the backbone of lattice QCD with better software. So instead they keep it internal. It's a bit sad that there's so little glory in writing better numerical libraries, but it's a common problem across the sciences (and in the open source community in general) so I can believe they'd be reluctant to share.