3 ms·

from the documentation: "You should expect 4 MB/s of throughput per TB of data stored as Nearline Storage. This throughput scales linearly with increased storag

by nda 12y ago

from the documentation: "You should expect 4 MB/s of throughput per TB of data stored as Nearline Storage. This throughput scales linearly with increased storage consumption."

blfr 12y ago
There it is. Still, someone like 'cperciva could pool a lot of their customers' data together and offer high speeds even for small amounts of data.
- gearhart 12y ago
  http://xkcd.com/1494/ http://xkcd.com/1494/ The unusual thing about this market is that by constantly improving the price point of their product but keeping their profit margins the same, AWS has forced every competitor in it to actually compete at the limits of their capability - so any scheme like this that got any serious traction would presumably be self-undermining. The economists will have considered all of the unused provided capacity in the model before they priced it. Sorry to be miserable ;)
  - vidarh 12y ago
    I don't know why you think so. AWS is not very price competitive. I make good money consulting on setup of more cost-effective alternatives to things like S3. The large cloud providers sell on brand recognition, convenience and trust, not cost. They may compete with each other on cost to try to cannibalise each others markets, but they're not even close to pushing the envelope on cost efficiency in terms of the prices they offer.
    - gearhart 12y ago
      Interesting - I'm speaking from general market sentiment (e.g. http://www.techrepublic.com/article/aws-is-playing-chicken-with-pricing-and-enterprise-incumbents-will-lose/ http://www.techrepublic.com/article/aws-is-playing-chicken-w...) but I'd never heard the "Mountain of Margin" argument (http://www.forbes.com/sites/mikekavis/2014/08/08/how-niche-cloud-providers-compete-with-aws-google-and-microsoft/ http://www.forbes.com/sites/mikekavis/2014/08/08/how-niche-c...) before you called it in to question - thanks for bringing it up. I now don't know what to believe - time to go and do my homework on this one.
    - gaadd33 12y ago
      Just wondering, what is more cost effective than S3 with the same reliability? If you have your own servers I can see running zfs and replicating snapshots every few minutes to a remote machine but if you aren't, what else is competitive? I do agree that a lot of AWS services aren't cost effective though, especially at any sort of non trivial scale.
      - vgt 12y ago
        Google Cloud Storage is cheaper than Amazon S3: http://aws.amazon.com/s3/pricing/ http://aws.amazon.com/s3/pricing/ https://cloud.google.com/storage/pricing https://cloud.google.com/storage/pricing
        vidarh 12y ago
        Google Cloud Storage is also massively overpriced.
        wstrange 12y ago
        Compared to what? Honest question - not trying to be cheeky. If you are comparing against a self managed solution, are you factoring all costs into that equation (fully burdened labour costs, disposal, cooling, power, etc.).
      - vidarh 12y ago
        Setting up your own with any of Ceph, Gluster, Riak, Swift or similar depending on your specific needs. Heck, I've got setups where we've been served well for years with a combination of inotifywait, grep and rsync (combined to trigger rsync instantly on modification events). It really depends on your access patterns, and there are lots of potential savings from making use of domain specific knowledge for your specific system. In general, you can beat AWS with 3x replication across multiple data centres even with renting managed servers, especially as your bandwidth use grows as AWS bandwidth prices are absolutely ridiculous (as in, anything from a factor of 5 to 20 above what you'll pay if you shop around and depending on your other requirements). Lease to own in a colo drops the price even further. Anything from 1/3 to 1/2 of AWS costs is reasonable with relatively moderate bandwidth usage, with the cost differential generally increasing substantially the more you access the data. Most people also don't have uniform storage needs. If you use your own setup, people tend to be able to cut substantially more in cost by reducing redundancy for data where it's not necessary etc. See also the Backblaze article that mentioned Reed-Solomon - if your app is suitable for doing similar you can totally blow the S3 costs further out of the water that way.
  - wongarsu 12y ago
    For a reference point: The hardware cost for storage servers with full redundancy is in the order of $0.10 per GB (plus electricity, rack space etc.). Buying your own hardware and throwing it away every year is price competitive with Nearline (and you get full online storage, no three second delays and slow retrieval). Of course there's good reasons not to roll your own solution until you reach a certain scale, but it's certainly possible to compete with Google and AWS on price.
    - dragonwriter 12y ago
      > For a reference point: The hardware cost for storage servers with full redundancy is in the order of $0.10 per GB (plus electricity, rack space etc.). Buying your own hardware and throwing it away every year is price competitive with Nearline (and you get full online storage, no three second delays and slow retrieval). Well, it would be price competitive if you considered only the hardware costs and not, e.g., the labor cost involved.
      - vidarh 12y ago
        Consider that there are labor costs around operating services on AWS as well, and it's generally more expensive by the hour, and labor cost of managing your own servers is fairly low. I have plenty of servers sitting in racks that haven't been touched in 5+ years; if you were to operate on a "buy and throw out in a year" practice, my typical labour costs would average a couple of hours per new server for setup. Or if you rent managed servers, you don't ever need to touch (or see) the hardware. Conversely, you can get bandwidth at a tiny fraction of the cost of AWS bandwidth, so the moment you actually transfer data to/from your setup, AWS gets progressively more expensive.
keysheath 12y ago
That would mean that if you had about 1TB stored it would take more than 3 days to retrieve it (with an initial 3 second delay before it starts).
- deleted 12y ago
  [deleted]
- wongarsu 12y ago
  And since bandwidth scales linearly with used storage it would presumably always take more than 3 days to retrieve all of your data.
- xhrpost 12y ago
  That's a good observation. I wonder how it works for multiple objects that are significantly smaller than 1TB. If I'm streaming one at 4MBps and then after a bit, decide to start another object download, does the original slow to 2MBps?
deleted 12y ago
[deleted]

by nda 12y ago

from the documentation: "You should expect 4 MB/s of throughput per TB of data stored as Nearline Storage. This throughput scales linearly with increased storage consumption."

blfr 12y ago

There it is. Still, someone like 'cperciva could pool a lot of their customers' data together and offer high speeds even for small amounts of data.

gearhart 12y ago
http://xkcd.com/1494/ http://xkcd.com/1494/ The unusual thing about this market is that by constantly improving the price point of their product but keeping their profit margins the same, AWS has forced every competitor in it to actually compete at the limits of their capability - so any scheme like this that got any serious traction would presumably be self-undermining. The economists will have considered all of the unused provided capacity in the model before they priced it. Sorry to be miserable ;)
- vidarh 12y ago
  I don't know why you think so. AWS is not very price competitive. I make good money consulting on setup of more cost-effective alternatives to things like S3. The large cloud providers sell on brand recognition, convenience and trust, not cost. They may compete with each other on cost to try to cannibalise each others markets, but they're not even close to pushing the envelope on cost efficiency in terms of the prices they offer.
  - gearhart 12y ago
    Interesting - I'm speaking from general market sentiment (e.g. http://www.techrepublic.com/article/aws-is-playing-chicken-with-pricing-and-enterprise-incumbents-will-lose/ http://www.techrepublic.com/article/aws-is-playing-chicken-w...) but I'd never heard the "Mountain of Margin" argument (http://www.forbes.com/sites/mikekavis/2014/08/08/how-niche-cloud-providers-compete-with-aws-google-and-microsoft/ http://www.forbes.com/sites/mikekavis/2014/08/08/how-niche-c...) before you called it in to question - thanks for bringing it up. I now don't know what to believe - time to go and do my homework on this one.
  - gaadd33 12y ago
    Just wondering, what is more cost effective than S3 with the same reliability? If you have your own servers I can see running zfs and replicating snapshots every few minutes to a remote machine but if you aren't, what else is competitive? I do agree that a lot of AWS services aren't cost effective though, especially at any sort of non trivial scale.
    - vgt 12y ago
      Google Cloud Storage is cheaper than Amazon S3: http://aws.amazon.com/s3/pricing/ http://aws.amazon.com/s3/pricing/ https://cloud.google.com/storage/pricing https://cloud.google.com/storage/pricing
      - vidarh 12y ago
        Google Cloud Storage is also massively overpriced.
        wstrange 12y ago
        Compared to what? Honest question - not trying to be cheeky. If you are comparing against a self managed solution, are you factoring all costs into that equation (fully burdened labour costs, disposal, cooling, power, etc.).
    - vidarh 12y ago
      Setting up your own with any of Ceph, Gluster, Riak, Swift or similar depending on your specific needs. Heck, I've got setups where we've been served well for years with a combination of inotifywait, grep and rsync (combined to trigger rsync instantly on modification events). It really depends on your access patterns, and there are lots of potential savings from making use of domain specific knowledge for your specific system. In general, you can beat AWS with 3x replication across multiple data centres even with renting managed servers, especially as your bandwidth use grows as AWS bandwidth prices are absolutely ridiculous (as in, anything from a factor of 5 to 20 above what you'll pay if you shop around and depending on your other requirements). Lease to own in a colo drops the price even further. Anything from 1/3 to 1/2 of AWS costs is reasonable with relatively moderate bandwidth usage, with the cost differential generally increasing substantially the more you access the data. Most people also don't have uniform storage needs. If you use your own setup, people tend to be able to cut substantially more in cost by reducing redundancy for data where it's not necessary etc. See also the Backblaze article that mentioned Reed-Solomon - if your app is suitable for doing similar you can totally blow the S3 costs further out of the water that way.
- wongarsu 12y ago
  For a reference point: The hardware cost for storage servers with full redundancy is in the order of $0.10 per GB (plus electricity, rack space etc.). Buying your own hardware and throwing it away every year is price competitive with Nearline (and you get full online storage, no three second delays and slow retrieval). Of course there's good reasons not to roll your own solution until you reach a certain scale, but it's certainly possible to compete with Google and AWS on price.
  - dragonwriter 12y ago
    > For a reference point: The hardware cost for storage servers with full redundancy is in the order of $0.10 per GB (plus electricity, rack space etc.). Buying your own hardware and throwing it away every year is price competitive with Nearline (and you get full online storage, no three second delays and slow retrieval). Well, it would be price competitive if you considered only the hardware costs and not, e.g., the labor cost involved.
    - vidarh 12y ago
      Consider that there are labor costs around operating services on AWS as well, and it's generally more expensive by the hour, and labor cost of managing your own servers is fairly low. I have plenty of servers sitting in racks that haven't been touched in 5+ years; if you were to operate on a "buy and throw out in a year" practice, my typical labour costs would average a couple of hours per new server for setup. Or if you rent managed servers, you don't ever need to touch (or see) the hardware. Conversely, you can get bandwidth at a tiny fraction of the cost of AWS bandwidth, so the moment you actually transfer data to/from your setup, AWS gets progressively more expensive.

keysheath 12y ago

That would mean that if you had about 1TB stored it would take more than 3 days to retrieve it (with an initial 3 second delay before it starts).

deleted 12y ago
[deleted]
wongarsu 12y ago
And since bandwidth scales linearly with used storage it would presumably always take more than 3 days to retrieve all of your data.
xhrpost 12y ago
That's a good observation. I wonder how it works for multiple objects that are significantly smaller than 1TB. If I'm streaming one at 4MBps and then after a bit, decide to start another object download, does the original slow to 2MBps?

deleted 12y ago

[deleted]