7 ms·

I do genome sequencing and this is very true. You are either all-in on cloud or do the processing completely in house. The data egress fees are more than enough

I do genome sequencing and this is very true. You are either all-in on cloud or do the processing completely in house. The data egress fees are more than enough to offset any benefit to a hybrid approach. I’m more curious if there is an equivalent to S3 for on-prem. I bet not due to reliability constraints, but something like this would be nice. I can’t say there would be much benefit to using AWS Outposts for genome sequencing without an S3 local cache.

matt_heimer 5y ago
If you are asking if Outputs includes S3, yes, https://aws.amazon.com/s3/outposts/ https://aws.amazon.com/s3/outposts/ If you are asking if there are stand-alone S3 API offerings, you might want to look at MinIO. Reliability mostly requires extra hardware and configuration - https://docs.min.io/docs/minio-erasure-code-quickstart-guide.html https://docs.min.io/docs/minio-erasure-code-quickstart-guide...
- mbreese 5y ago
  I didn't know they also had S3 for outposts... that's interesting. That also requires an Outposts rack as opposed to the servers the original article was talking about. > 380 TB S3 tier is priced at $0.1000/GB-mo ($38,912.00/month) But it's also extremely expensive. $38K per month for 380 TB of storage. Even if we assume 3X redundancy, you can get 500TB of raw capacity (12-16TB HDDs) for ~ $25k. After 3 months of usage, you'd have already covered that initial expense. I know there are other benefits to having managed services, but again, cost is definitely prohibitive for many use-cases. I use minio locally to support S3-workloads on our local cluster. We can then migrate workloads to AWS/S3 proper with minimal changes, if needed.
r00fus 5y ago
How do you even get the petabytes of data into S3 without incurring massive charges? Previous estimations of getting the initial dataset into cloud was.. prohibitive (note this was 3-4 years ago).
- laurencerowe 5y ago
  What charges in particular are you referring to? The cloud services only charge data egress fees. Data ingress is free. The ENCODE Project data were a hundred terabytes or so at the point I migrated it into S3 (it's now over a petabyte.) We were on a fast academic network with a 100GB interconnect so could just `aws s3 sync` rather than worry about using AWS Snowballs or anything.
- nicornk 5y ago
  Uploading data into AWS is free.

matt_heimer 5y ago

If you are asking if Outputs includes S3, yes, https://aws.amazon.com/s3/outposts/ https://aws.amazon.com/s3/outposts/ If you are asking if there are stand-alone S3 API offerings, you might want to look at MinIO. Reliability mostly requires extra hardware and configuration - https://docs.min.io/docs/minio-erasure-code-quickstart-guide.html https://docs.min.io/docs/minio-erasure-code-quickstart-guide...

mbreese 5y ago
I didn't know they also had S3 for outposts... that's interesting. That also requires an Outposts rack as opposed to the servers the original article was talking about. > 380 TB S3 tier is priced at $0.1000/GB-mo ($38,912.00/month) But it's also extremely expensive. $38K per month for 380 TB of storage. Even if we assume 3X redundancy, you can get 500TB of raw capacity (12-16TB HDDs) for ~ $25k. After 3 months of usage, you'd have already covered that initial expense. I know there are other benefits to having managed services, but again, cost is definitely prohibitive for many use-cases. I use minio locally to support S3-workloads on our local cluster. We can then migrate workloads to AWS/S3 proper with minimal changes, if needed.

r00fus 5y ago

How do you even get the petabytes of data into S3 without incurring massive charges? Previous estimations of getting the initial dataset into cloud was.. prohibitive (note this was 3-4 years ago).

laurencerowe 5y ago
What charges in particular are you referring to? The cloud services only charge data egress fees. Data ingress is free. The ENCODE Project data were a hundred terabytes or so at the point I migrated it into S3 (it's now over a petabyte.) We were on a fast academic network with a 100GB interconnect so could just `aws s3 sync` rather than worry about using AWS Snowballs or anything.
nicornk 5y ago
Uploading data into AWS is free.