4 ms·

I've never worked in this space, but I'm curious about the need for massive egress. What's driving the need to bring all that data back to the institution? Cou

I've never worked in this space, but I'm curious about the need for massive egress. What's driving the need to bring all that data back to the institution? Could whatever actions have to be performed on the data also be performed in AWS? Also while briefly looking into this I found that AWS has an egress waiver for researchers and educational instiutions: https://aws.amazon.com/blogs/publicsector/data-egress-waiver-available-for-eligible-researchers-and-institutions/ https://aws.amazon.com/blogs/publicsector/data-egress-waiver...

COGlory 4y ago
Well for starters, if you are NIH or NSF funded, they have data storage requirements you must meet. So usually this involves something like tape backups in two locations. The other is for reproducibility - typically you need to preserve lots of in-between steps for peer review and proving that you aren't making things up. Some intermediary data is wiped out, but usually only if it can be quickly and easily regenerated.
jpeloquin 4y ago
Regarding the waiver—"The maximum discount is 15 percent of total monthly spending on AWS services". Was very excited at first. As for leaving data in AWS, data is often (not always) revisited repeatedly for years after the fact. If new questions are raised about the results it's often much easier to check the output than rerun the analysis. And cloud storage is not cheap. But yes it sometimes makes sense to egress only summary statistics and discard the raw data.
jltsiren 4y ago
I've worked the last few years in a research institute well-funded by academic standards. During that time, people around me have used self-hosted hardware, at least three different cloud providers, and HPC clusters hosted by several different organizations. I've also worked with people from many other institutions, each of which has made its own infrastructure choices. You may use AWS yourself, but your collaborators have made their own choices for their own reasons. The data goes where it's needed, often crossing institutional boundaries.

COGlory 4y ago

Well for starters, if you are NIH or NSF funded, they have data storage requirements you must meet. So usually this involves something like tape backups in two locations. The other is for reproducibility - typically you need to preserve lots of in-between steps for peer review and proving that you aren't making things up. Some intermediary data is wiped out, but usually only if it can be quickly and easily regenerated.

jpeloquin 4y ago

Regarding the waiver—"The maximum discount is 15 percent of total monthly spending on AWS services". Was very excited at first. As for leaving data in AWS, data is often (not always) revisited repeatedly for years after the fact. If new questions are raised about the results it's often much easier to check the output than rerun the analysis. And cloud storage is not cheap. But yes it sometimes makes sense to egress only summary statistics and discard the raw data.

jltsiren 4y ago

I've worked the last few years in a research institute well-funded by academic standards. During that time, people around me have used self-hosted hardware, at least three different cloud providers, and HPC clusters hosted by several different organizations. I've also worked with people from many other institutions, each of which has made its own infrastructure choices. You may use AWS yourself, but your collaborators have made their own choices for their own reasons. The data goes where it's needed, often crossing institutional boundaries.