26 ms·
I work in particle astrophysics data processing and management, mostly with gamma ray events for Fermi gamma ray space-telescope. Our total data size after five
by batbomb 13y ago
I work in particle astrophysics data processing and management, mostly with gamma ray events for Fermi gamma ray space-telescope. Our total data size after five years is somewhere around 2 PB, but the data set usually used for analysis runs about 200 TB, not really that large. AFAIK, none of the 100+ collaborators use Hadoop. Mostly they use software I develop to run batch jobs in parallel on one of several clusters.
LSST, which I also am starting to work on, will be different, as the raw data alone will approach 4TB a day.
- dalke 13y agoCERN collects about 25 PB/year.
- awhitty 13y agoFor some reason, wrapping my mind around this is like trying to fathom the Grand Canyon or the scale of the universe.
- dalke 13y agoYou and me both. It's made even worse when you learn that the first-level trigger removes 90% of the input stream before passing it to 2,000 computers, which in turn select only 0.2% of that data for storage and further analysis. https://lhcb-public.web.cern.ch/lhcb-public/en/Data%20Collection/Triggers-en.html https://lhcb-public.web.cern.ch/lhcb-public/en/Data%20Collec...
- toomuchtodo 13y agoFermilab in Batavia takes data from CERN over a 40Gb/s optical link (CMS Tier1, CERN CMS is Tier0); when I was there several years ago, we were staging the data to ~500TB of spinning disk in Nexans and cold storage were 2-3 Storagetek tape libraries the size of school busses. Computing and particle physics is where awesome meet.
- dekhn 13y agoCERN and its associated teams (CMS and ATLAS) are awesome. They are the only outfit outside of Google that seems to be handling data management and computing properly.
- toomuchtodo 13y agoNot to disappoint you, but I could tell you horror stories about my time working on the CMS team at Fermilab. Horrible, horrible stories. I hope the CMS and ATLAS teams onsite at CERN were much better.
- ohashi 13y agoI would be curious to hear some of these stories.
- toomuchtodo 13y agoI'll have to put a blog post together about it. Extremely poor planning, no accountability, a boss who was verbally and emotionally abusive to a co-worker (the reason I quit after only a year there and went back to the private sector).
- cshimmin 13y agoATLAS guy here. I agree with @toomuchtodo, we're stuck in the stone age. It's basically just batch processing and lots of copying data around for no reason.
- dekhn 13y agoYou could say the same thing about google's backend (batch processing and copying data), but it seems to be working for them as well.
- porges 13y agoIn the future, the Square Kilometre Array is forecast to require 300-1500 PB/year of archiving, somewhere between 10 PB/hr and 1 EB/day of raw data. (That's 3-12 TB/s...)