4 ms·

Off on a tangent, but does anybody know why they store the data from the radio telescope array for such a long time? Wouldn't it be more efficient to analyze i

Off on a tangent, but does anybody know why they store the data from the radio telescope array for such a long time? Wouldn't it be more efficient to analyze it, save the interesting data, and get rid of the rest?

schiffern 14y ago
Lots of discoveries in astronomy are now made by re-analyzing data from previous observations. The data storage costs are a pittance compared to the cost of telescope time. By saving the data you're getting the most out of each observation.
bithive123 14y ago
I think you answered your own question. In many cases finding out which data is interesting _is_ the analysis.
andrewcooke 14y ago
am not sure re-analysis is that important myself, but it looks like the SKA will be used in a pretty traditional way, with groups of astronomers applying to do particular observations. that means that the processing of data will likely be different for each group, and will probably take place over time (some poor grad student will likely spend most of their thesis - several years - on it). the other approach - taken with something like the SDSS - is to make a telescope for just one task (typically a survey). then you can process the data and throw it away. i am pretty sure that is what SDSS did, and what, say, the LSST will probably do too (if it ever gets finished). the advantage of the first approach is that you are much more flexible, which means more likelihood of making a big discovery (particularly when, as with this telescope, collecting area is larger than ever before, which means that you can see fainter and further back in time, making it ideal to study isolated, unusual objects - in contrast, survey telescopes make different technical compromises so that they cover a wider field of view), and also more chances to make and exploit upgrades over time. the downside is dealing with issues like this (another issue is data transport - traditionally you go to a telescope and then take the data home with you; i imagine we're now getting to the point where instead you will process data local to the data storage). disclaimer: i was an optical astronomer, not radio, and that was years ago, so this may already be old news / incorrect in details. but the general idea should be ok. http://en.wikipedia.org/wiki/Square_Kilometre_Array http://en.wikipedia.org/wiki/Square_Kilometre_Array http://www.sdss.org/ http://www.sdss.org/ http://www.lsst.org/lsst/ http://www.lsst.org/lsst/ ps often telescopes do make data public after a certain time. but the idea is not so much to allow reanalysis as to make sure the people who originally took the data reduce and publish it. it's easy to postpone that kind of work, but the idea that someone else might do it and publish first is quite a motivator.
- schiffern 14y ago
  SDSS raw data is available online: http://www.sdss.org/dr7/algorithms/dataProcessing.html http://www.sdss.org/dr7/algorithms/dataProcessing.html
acdha 14y ago
Science is all about reproducibility: if you discard the original data and someone raises a question about your processing techniques, you face the prospect of having to completely retract everything which depends on that data. Entire careers are potentially at risk and the failure mode might not even be “spend a year or two repeating an experiment” (i.e. falling behind everyone else in your field) but “hope desperately that some rare event occurs again”. There's also a long history of data being mined later for things which the original experimenters didn't even think to collect - e.g. http://www.nasa.gov/mission_pages/hubble/science/elusive-planets.html http://www.nasa.gov/mission_pages/hubble/science/elusive-pla... - or types of analysis which was considered too expensive to perform in the past (e.g. biology went from a pre- to post- informatics era when the unit of work went from “single grad student” to “many-thousand node cluster”). Given how expensive the data was to collect, it almost always makes sense to see if it will lead to any other value.

schiffern 14y ago

Lots of discoveries in astronomy are now made by re-analyzing data from previous observations. The data storage costs are a pittance compared to the cost of telescope time. By saving the data you're getting the most out of each observation.

bithive123 14y ago

I think you answered your own question. In many cases finding out which data is interesting _is_ the analysis.

andrewcooke 14y ago

am not sure re-analysis is that important myself, but it looks like the SKA will be used in a pretty traditional way, with groups of astronomers applying to do particular observations. that means that the processing of data will likely be different for each group, and will probably take place over time (some poor grad student will likely spend most of their thesis - several years - on it). the other approach - taken with something like the SDSS - is to make a telescope for just one task (typically a survey). then you can process the data and throw it away. i am pretty sure that is what SDSS did, and what, say, the LSST will probably do too (if it ever gets finished). the advantage of the first approach is that you are much more flexible, which means more likelihood of making a big discovery (particularly when, as with this telescope, collecting area is larger than ever before, which means that you can see fainter and further back in time, making it ideal to study isolated, unusual objects - in contrast, survey telescopes make different technical compromises so that they cover a wider field of view), and also more chances to make and exploit upgrades over time. the downside is dealing with issues like this (another issue is data transport - traditionally you go to a telescope and then take the data home with you; i imagine we're now getting to the point where instead you will process data local to the data storage). disclaimer: i was an optical astronomer, not radio, and that was years ago, so this may already be old news / incorrect in details. but the general idea should be ok. http://en.wikipedia.org/wiki/Square_Kilometre_Array http://en.wikipedia.org/wiki/Square_Kilometre_Array http://www.sdss.org/ http://www.sdss.org/ http://www.lsst.org/lsst/ http://www.lsst.org/lsst/ ps often telescopes do make data public after a certain time. but the idea is not so much to allow reanalysis as to make sure the people who originally took the data reduce and publish it. it's easy to postpone that kind of work, but the idea that someone else might do it and publish first is quite a motivator.

schiffern 14y ago
SDSS raw data is available online: http://www.sdss.org/dr7/algorithms/dataProcessing.html http://www.sdss.org/dr7/algorithms/dataProcessing.html

acdha 14y ago

Science is all about reproducibility: if you discard the original data and someone raises a question about your processing techniques, you face the prospect of having to completely retract everything which depends on that data. Entire careers are potentially at risk and the failure mode might not even be “spend a year or two repeating an experiment” (i.e. falling behind everyone else in your field) but “hope desperately that some rare event occurs again”. There's also a long history of data being mined later for things which the original experimenters didn't even think to collect - e.g. http://www.nasa.gov/mission_pages/hubble/science/elusive-planets.html http://www.nasa.gov/mission_pages/hubble/science/elusive-pla... - or types of analysis which was considered too expensive to perform in the past (e.g. biology went from a pre- to post- informatics era when the unit of work went from “single grad student” to “many-thousand node cluster”). Given how expensive the data was to collect, it almost always makes sense to see if it will lead to any other value.