4 ms·

I do wonder what the best form of compression would be and given there web pages if some form of customed compression that was optimised for HTML would be usefu

by Zenst 14y ago

I do wonder what the best form of compression would be and given there web pages if some form of customed compression that was optimised for HTML would be useful. There again with that volume of data, what would CERN do for storage/access for a data pool that size and still be useable. Reason being if you wanted to back that lot of up ship copies for research purposes then with todays technology the humble memory stick, even the biggest would fail to even handle the file index on this scale. Scary amount of data. But certainly a data set many would like to play with and try things out, being the geeks we are.

juiceandjuice 14y ago
CERN has a variety of techniques to access, store, and backup their data. I know from experience some experiments, or at least some parts of experiments, use layers and layers of abstraction, like scalla/xrootd which operates on clusters of servers directly or with nfs. In addition to this, there's levels of processed data. For example, raw data is usually level 0, and basic processed data is level 1 Processed and calibrated data is usually level 2, etc... but experiments often have different definitions for each level. Reprocessing of any later can happen at any time, although level 1 reprocessing is usually an extremely intensive operation because it operates with the largest amount of data. Level 0 data is usually heavily compressed when it's left on disk, because it's typically the largest amount of data, but also least touched. Most scientists will use level 2 or level 1 data. This data will be on low latency clusters. So, while CERN has petabytes of data, typically a fraction of that will be accessible. In the past, level 0 data was often left to tape. While the raw data is still backed up to tape (I know this is the case for ATLAS), for many experiments with large amounts of data they might leave it on lower cost HDD in simple RAID arrays for redundancy and not worry about performance so much. The BaBar experiment has done this for their long term data analysis. In addition to all of this, it's still occasionally easier to transfer large amounts of data via tape instead of the internet.
mmagin 14y ago
Back when I was involved, the compression was gzip -- each archived page (HTTP headers + body) was separately gzipped and concatenated onto a ~100M file, though I believe they're now doing 1G files. (This cleverly allows retrieving the individual document by seeking to that offset and doing gzip decompression.) They have significant amounts of inline images and other files in the web archive, which of course typically are already compressed. http://en.wikipedia.org/wiki/Web_ARChive http://en.wikipedia.org/wiki/Web_ARChive

by Zenst 14y ago

juiceandjuice 14y ago

CERN has a variety of techniques to access, store, and backup their data. I know from experience some experiments, or at least some parts of experiments, use layers and layers of abstraction, like scalla/xrootd which operates on clusters of servers directly or with nfs. In addition to this, there's levels of processed data. For example, raw data is usually level 0, and basic processed data is level 1 Processed and calibrated data is usually level 2, etc... but experiments often have different definitions for each level. Reprocessing of any later can happen at any time, although level 1 reprocessing is usually an extremely intensive operation because it operates with the largest amount of data. Level 0 data is usually heavily compressed when it's left on disk, because it's typically the largest amount of data, but also least touched. Most scientists will use level 2 or level 1 data. This data will be on low latency clusters. So, while CERN has petabytes of data, typically a fraction of that will be accessible. In the past, level 0 data was often left to tape. While the raw data is still backed up to tape (I know this is the case for ATLAS), for many experiments with large amounts of data they might leave it on lower cost HDD in simple RAID arrays for redundancy and not worry about performance so much. The BaBar experiment has done this for their long term data analysis. In addition to all of this, it's still occasionally easier to transfer large amounts of data via tape instead of the internet.

mmagin 14y ago

Back when I was involved, the compression was gzip -- each archived page (HTTP headers + body) was separately gzipped and concatenated onto a ~100M file, though I believe they're now doing 1G files. (This cleverly allows retrieving the individual document by seeking to that offset and doing gzip decompression.) They have significant amounts of inline images and other files in the web archive, which of course typically are already compressed. http://en.wikipedia.org/wiki/Web_ARChive http://en.wikipedia.org/wiki/Web_ARChive