5 ms·

The problem with AI in particle physics is that you need to very well understand the efficiency of the selection ( rate of false negative and false positive cla

The problem with AI in particle physics is that you need to very well understand the efficiency of the selection ( rate of false negative and false positive classification). And this rate of course isn't uniform for all kinds of events. Thus, the uncertainty on the selection efficiency tends to grow with the complexity of the machine learning model. This directly hurts your sensitivy, the very thing you were trying to improve by using an AI - based process. It's a trade-off.

jessriedel 9y ago
From what I remember doing this years ago, the selection efficiency is usually determined through numerical simulation rather than analytically. So why would your estimate of the efficiency get worse with more complicated models?
- dukwon 9y ago
  In general, simulation doesn't agree with data perfectly. The more the variables (and their correlations) aren't reproduced faithfully, the more the efficiency determined from simulation is likely to depart from the truth. It's even possible to accidentally train an algortihm to discriminate between simulation and data rather than signal and background. It's common practice to check data/simulation agreements between variables before using them for training. Reweighting procedures are used to improve agreement.
Analog24 9y ago
As already pointed out, detector efficiency is determined through extremely detailed simulations of the entire system. Those measurements are done in a completely orthogonal manner so the complexity of the model has nothing to do with the systematic uncertainties that arise from the selection efficiency.
- konschubert 9y ago
  > As already pointed out, detector efficiency is determined through extremely detailed simulations of the entire system. It depends on the experiment, LHCb for example does not use simulated background. In any case, the more complex your model gets (number of variables) the exponentially more simulated Monte Carlo events you need to fill that multidimensional space.
  - Analog24 9y ago
    I'm not talking about simulating background. Every experiment uses simulation (an extremely accurate and well understood package called GEANT) to simulate the detector response to measure efficiencies. You're absolutely right about how the amount of data needed for statistical significance grows with the size of the parameter space you're searching.
  - dukwon 9y ago
    > LHCb for example does not use simulated background. That depends on the analysis. If you're looking at a partially-reconstructed decay (common in semileptonics) then you can't rely on the regular trick of choosing a sideband sample to work as your combinatorial background. Also, it's very common to model specific misidentified or partially-reconstructed backgrounds using simulation. > In any case, the more complex your model gets (number of variables) the exponentially more simulated Monte Carlo events you need to fill that multidimensional space. I understand this argument if you're trying to model the efficiency in nD space (through splines, histograms, moments etc), but that's usually done when you're fitting to n variables, e.g. in an amplitude fit. If you just want the efficiency of a cut on the score from an MVA algorithm, I don't think it matters. What definitely matters is that the behaviour on simulation reproduces that of real signal as faithfully as possible.
    - konschubert 9y ago
      > I understand this argument if you're trying to model the efficiency in nD space (through splines, histograms, moments etc), but that's usually done when you're fitting to n variables, e.g. in an amplitude fit. If you just want the efficiency of a cut on the score from an MVA algorithm, I don't think it matters. What definitely matters is that the behaviour on simulation reproduces that of real signal as faithfully as possible. You might have a point there.

jessriedel 9y ago

From what I remember doing this years ago, the selection efficiency is usually determined through numerical simulation rather than analytically. So why would your estimate of the efficiency get worse with more complicated models?

dukwon 9y ago
In general, simulation doesn't agree with data perfectly. The more the variables (and their correlations) aren't reproduced faithfully, the more the efficiency determined from simulation is likely to depart from the truth. It's even possible to accidentally train an algortihm to discriminate between simulation and data rather than signal and background. It's common practice to check data/simulation agreements between variables before using them for training. Reweighting procedures are used to improve agreement.

Analog24 9y ago

As already pointed out, detector efficiency is determined through extremely detailed simulations of the entire system. Those measurements are done in a completely orthogonal manner so the complexity of the model has nothing to do with the systematic uncertainties that arise from the selection efficiency.

konschubert 9y ago
> As already pointed out, detector efficiency is determined through extremely detailed simulations of the entire system. It depends on the experiment, LHCb for example does not use simulated background. In any case, the more complex your model gets (number of variables) the exponentially more simulated Monte Carlo events you need to fill that multidimensional space.
- Analog24 9y ago
  I'm not talking about simulating background. Every experiment uses simulation (an extremely accurate and well understood package called GEANT) to simulate the detector response to measure efficiencies. You're absolutely right about how the amount of data needed for statistical significance grows with the size of the parameter space you're searching.
- dukwon 9y ago
  > LHCb for example does not use simulated background. That depends on the analysis. If you're looking at a partially-reconstructed decay (common in semileptonics) then you can't rely on the regular trick of choosing a sideband sample to work as your combinatorial background. Also, it's very common to model specific misidentified or partially-reconstructed backgrounds using simulation. > In any case, the more complex your model gets (number of variables) the exponentially more simulated Monte Carlo events you need to fill that multidimensional space. I understand this argument if you're trying to model the efficiency in nD space (through splines, histograms, moments etc), but that's usually done when you're fitting to n variables, e.g. in an amplitude fit. If you just want the efficiency of a cut on the score from an MVA algorithm, I don't think it matters. What definitely matters is that the behaviour on simulation reproduces that of real signal as faithfully as possible.
  - konschubert 9y ago
    > I understand this argument if you're trying to model the efficiency in nD space (through splines, histograms, moments etc), but that's usually done when you're fitting to n variables, e.g. in an amplitude fit. If you just want the efficiency of a cut on the score from an MVA algorithm, I don't think it matters. What definitely matters is that the behaviour on simulation reproduces that of real signal as faithfully as possible. You might have a point there.