4 ms·

I've yet to read the article, but I'll hazard a guess. At the time of LHC's design and instantiation, the amount of raw data it generated was vastly more than

I've yet to read the article, but I'll hazard a guess. At the time of LHC's design and instantiation, the amount of raw data it generated was vastly more than could be stored and arbitrarily processed. So, it was quickly filtered and shunted into processes designed to look for certain things. Communications and processing capabilities have increased substantially since then. One may now be able, with investment/expansion of those, to take a more "brute force" approach in mining closer to if not at the level of raw data and arbitrary -- or at least a broader range of -- associations.

Analog24 8y ago
It's not quite that simple. The CMS detector (one of the two main detectors/experiments operating at the LHC) spits out about 10TB of raw data per second. This is more than any system can handle at the moment. To deal with this overwhelming flood of data there are layers of "triggers" that effectively filter out the collisions that aren't of any interest. The lowest level triggers are actually embedded in FPGAs on the detector itself, then there is a high-level trigger that runs on an on-site server farm that does minimal event processing to see if the even is interesting. All of this requires us to define what constituents an interesting event. The most basic trigger is called the "minimum bias trigger" because it requires us to make the fewest assumptions about what makes the event interesting. So even at the most general level a brute force approach will still have some bias because it will likely be using the minimum bias data. What makes it "brute force" is that they will no longer be looking for signals from specific new models, they will only be considering the Standard Model and essentially looking for outliers. The difficult part is that there is so much systematic uncertainty given the complexity of these detectors, which makes it hard to obtain statistical significance.
- hinkley 8y ago
  Oh! So we are using the term brute force in the information theory sense of the word, not the scientific connotation?
- DoctorOetker 8y ago
  could you elaborate a bit on the "10 TB of raw data per second" ? If it's really raw how much room is there for domain specific lossless compression? How many seconds long has or will the experiment (CMS) normally run? How much did or will the CMS component of LHC cost from construction to its full lifespan? What fraction of that imaginary lump sum went to digital storage budget?
  - Analog24 8y ago
    These experiments typically run 9 months out of the year with (very roughly) 50% up time. That's (again, very roughly ) 25,000,000 seconds. That's 250000 petabytes a year [0]. And by "raw" I'm talking about the ADC signals from the detector hardware. For example, the readout from the 75 megapixel "camera" (i.e. the tracker) at the core of the detector that takes 40 million pictures a second. When doing analysis the collisions are reconstructed into "events" where all the ADC signals are combined to form particle tracks and energy deposits (allowing us to actually study the physics behind the high energy interactions). These reconstruction algorithms are very complex and statistical in nature so there is some bias associated with them, hence the reason the "raw" data is permanently saved along with the reconstructed data. Compression is most definitely used for the data that is actually saved to disk but it is not done on the fly when trigger systems have to run in nanoseconds. The data limitations are not really from a lack of foresight or proper budgeting. It was known decades ago how much data would be produced by the LHC, it just really is an insane amount of data. [0] after applying triggers (i.e. filtering) only a few hundred petabytes of data is stored in a given year from the CMS detector.
    - toomuchtodo 8y ago
      It’s been ~10 years since I was at Fermilab working on the CMS Tier 1 team; when I was still new on the team, I expressed concern about data loss due to scheduled maintenance on our Nexsan clusters (coming from the private sector) and my boss would chuckle and say “Don’t worry, they’re always making more data”. Also, scheduled maintenance windows during the day! As an ops guy, I was spoiled for that period of time in my career.
    - 8y ago
- pasbesoin 8y ago
  Thank you. I find your explanation enlightening and interesting. I haven't spoken with him in some years, and I won't mention names, but a good friend worked on calibration of the CMS detectors (scintillation). The last time I did speak with him, that was a very interesting 2 or 3 hours. Although I don't recall us discussing the data processing aspects in any depth; my limited impression of those I mostly gained from random articles and whatnot. P.S. I guess I'd bettter RTFA, now. ;-)

Analog24 8y ago

It's not quite that simple. The CMS detector (one of the two main detectors/experiments operating at the LHC) spits out about 10TB of raw data per second. This is more than any system can handle at the moment. To deal with this overwhelming flood of data there are layers of "triggers" that effectively filter out the collisions that aren't of any interest. The lowest level triggers are actually embedded in FPGAs on the detector itself, then there is a high-level trigger that runs on an on-site server farm that does minimal event processing to see if the even is interesting. All of this requires us to define what constituents an interesting event. The most basic trigger is called the "minimum bias trigger" because it requires us to make the fewest assumptions about what makes the event interesting. So even at the most general level a brute force approach will still have some bias because it will likely be using the minimum bias data. What makes it "brute force" is that they will no longer be looking for signals from specific new models, they will only be considering the Standard Model and essentially looking for outliers. The difficult part is that there is so much systematic uncertainty given the complexity of these detectors, which makes it hard to obtain statistical significance.

hinkley 8y ago
Oh! So we are using the term brute force in the information theory sense of the word, not the scientific connotation?
DoctorOetker 8y ago
could you elaborate a bit on the "10 TB of raw data per second" ? If it's really raw how much room is there for domain specific lossless compression? How many seconds long has or will the experiment (CMS) normally run? How much did or will the CMS component of LHC cost from construction to its full lifespan? What fraction of that imaginary lump sum went to digital storage budget?
- Analog24 8y ago
  These experiments typically run 9 months out of the year with (very roughly) 50% up time. That's (again, very roughly ) 25,000,000 seconds. That's 250000 petabytes a year [0]. And by "raw" I'm talking about the ADC signals from the detector hardware. For example, the readout from the 75 megapixel "camera" (i.e. the tracker) at the core of the detector that takes 40 million pictures a second. When doing analysis the collisions are reconstructed into "events" where all the ADC signals are combined to form particle tracks and energy deposits (allowing us to actually study the physics behind the high energy interactions). These reconstruction algorithms are very complex and statistical in nature so there is some bias associated with them, hence the reason the "raw" data is permanently saved along with the reconstructed data. Compression is most definitely used for the data that is actually saved to disk but it is not done on the fly when trigger systems have to run in nanoseconds. The data limitations are not really from a lack of foresight or proper budgeting. It was known decades ago how much data would be produced by the LHC, it just really is an insane amount of data. [0] after applying triggers (i.e. filtering) only a few hundred petabytes of data is stored in a given year from the CMS detector.
  - toomuchtodo 8y ago
    It’s been ~10 years since I was at Fermilab working on the CMS Tier 1 team; when I was still new on the team, I expressed concern about data loss due to scheduled maintenance on our Nexsan clusters (coming from the private sector) and my boss would chuckle and say “Don’t worry, they’re always making more data”. Also, scheduled maintenance windows during the day! As an ops guy, I was spoiled for that period of time in my career.
  - 8y ago
pasbesoin 8y ago
Thank you. I find your explanation enlightening and interesting. I haven't spoken with him in some years, and I won't mention names, but a good friend worked on calibration of the CMS detectors (scintillation). The last time I did speak with him, that was a very interesting 2 or 3 hours. Although I don't recall us discussing the data processing aspects in any depth; my limited impression of those I mostly gained from random articles and whatnot. P.S. I guess I'd bettter RTFA, now. ;-)