3 ms·

This debate reminded me Matt Strassler's recent post that most of the data observed in the accelerators are thrown away [1]: So what’s to be done? There’s

by nyc111 2y ago

This debate reminded me Matt Strassler's recent post that most of the data observed in the accelerators are thrown away [1]: So what’s to be done? There’s only one option: throw most of that data away in the smartest way possible, and ensure that the data retained is processed and stored efficiently. I thought that was strange. It's like there is too much data and our technology is not up to it so let's throw away everything that we cannot process. Throwing data "in the smartest way possible" did not convince me. [1] https://profmattstrassler.com/2024/10/21/innovations-in-data-collection-at-the-lhc/ https://profmattstrassler.com/2024/10/21/innovations-in-data...

elashri 2y ago
I would like a chance to jump into this point because this problem is a function of two things. The throughput that you can make your Tigger (Data acquisition system) save the data and transfer it to permanent storage. This is usually invlove multiple steps and most of them happens in real-time. The other problem is the storage itself and how it would be kept (duplicated and distributed to analysts) which at the scale of operation we are doing is insanely costly. If we are to say save 20% of the generated collision data then we would fill the entire cloud storage in the world in a couple of runs . Also the vasr majority of data is background and useless so you would do a lot of work to clean that and apply your selections which we do anyway but now you are dealing with another problem. The analysts will need to handle much more data and trying new things (ideas and searches) becomes more costly which will be discouraged. So you work in a very constrained way. You improve your capabilities in computing and storage and you present a good physics case of what data (deploy trigger line which is to pick this physics signal) that the experiment is sensitive to and then lets the natural selections take place (metaphorically of course). Most of the experiments cannot because of the data acquisition problems.
dguest 2y ago
The technology really is not up to it, though. To give some numbers: - The LHC has 40M "events" (bunch of collisions) a second. - The experiments can afford to save around 2000 of them. This is a factor of 20k between what they collide and what they can afford to analyze. There is just no conceivable way to expand the LHC computing and storage by a factor of 20k. Valid question would be why they don't just collide fewer protons. The problem is that when you study processes on a length scale smaller than a proton, you really can't control when they happen. You just have to smash a lot and catch the interesting collisions. So yeah, it's a lot of "throwing away data" in the smartest way possible. ------------------- All that said, it might be a stretch to say the data is "thrown away", since that implies that it was ever acquired. The data that doesn't get saved generally doesn't make it off a memory buffer on a sensor deep within the detector. It's never piped through an actual CPU or assembled into any meaningful unit with the millions of other readouts. If keeping the data was one more trivial step, the experiments would keep it. As it is they need to be smart about where the attention goes. And they are! The data is "thrown away" in the sense that an astronomy experiment throws away data by turning off during the day.
- aeonik 2y ago
  My 200Mhz Oscilloscope "throws away" a lot of data compared to the 4GHz scope I used at work, but for the signals I'm looking for, it doesn't matter at all.

by nyc111 2y ago

elashri 2y ago

I would like a chance to jump into this point because this problem is a function of two things. The throughput that you can make your Tigger (Data acquisition system) save the data and transfer it to permanent storage. This is usually invlove multiple steps and most of them happens in real-time. The other problem is the storage itself and how it would be kept (duplicated and distributed to analysts) which at the scale of operation we are doing is insanely costly. If we are to say save 20% of the generated collision data then we would fill the entire cloud storage in the world in a couple of runs . Also the vasr majority of data is background and useless so you would do a lot of work to clean that and apply your selections which we do anyway but now you are dealing with another problem. The analysts will need to handle much more data and trying new things (ideas and searches) becomes more costly which will be discouraged. So you work in a very constrained way. You improve your capabilities in computing and storage and you present a good physics case of what data (deploy trigger line which is to pick this physics signal) that the experiment is sensitive to and then lets the natural selections take place (metaphorically of course). Most of the experiments cannot because of the data acquisition problems.

dguest 2y ago

The technology really is not up to it, though. To give some numbers: - The LHC has 40M "events" (bunch of collisions) a second. - The experiments can afford to save around 2000 of them. This is a factor of 20k between what they collide and what they can afford to analyze. There is just no conceivable way to expand the LHC computing and storage by a factor of 20k. Valid question would be why they don't just collide fewer protons. The problem is that when you study processes on a length scale smaller than a proton, you really can't control when they happen. You just have to smash a lot and catch the interesting collisions. So yeah, it's a lot of "throwing away data" in the smartest way possible. ------------------- All that said, it might be a stretch to say the data is "thrown away", since that implies that it was ever acquired. The data that doesn't get saved generally doesn't make it off a memory buffer on a sensor deep within the detector. It's never piped through an actual CPU or assembled into any meaningful unit with the millions of other readouts. If keeping the data was one more trivial step, the experiments would keep it. As it is they need to be smart about where the attention goes. And they are! The data is "thrown away" in the sense that an astronomy experiment throws away data by turning off during the day.

aeonik 2y ago
My 200Mhz Oscilloscope "throws away" a lot of data compared to the 4GHz scope I used at work, but for the signals I'm looking for, it doesn't matter at all.