7 ms·

As somebody who's been looking really hard at a project/side business that'd use Spaces (DO's object storage system), this makes me super, super nervous. To say

As somebody who's been looking really hard at a project/side business that'd use Spaces (DO's object storage system), this makes me super, super nervous. To say nothing of block storage--yikes. Can anyone speak to quality/reliability of other object storage providers that have S3-compatible (including presigned URL) APIs? S3's pricing is absolutely ridiculous by comparison, but they have the reliability argument on their side...

aedocw 9y ago
I would suggest you take a look at Wasabi for object storage. I'm just a customer, but have been using them for close to a year for off-site backup storage and it's been great.
- filleokus 9y ago
  Reposting from my comment from yesterday[0], how are the speeds from you location (and where's that)? It have been ridiculously slow from Northern Europe when I've tried it, like not even 1 MB/s down. 0: https://news.ycombinator.com/item?id=16726324 https://news.ycombinator.com/item?id=16726324
- eropple 9y ago
  Wasabi looks interesting. What's the catch? How's the outgoing bandwidth/download speed? (Large file downloads are my main concern.)
  - corobo 9y ago
    > Wasabi’s hot cloud storage service is not designed to be used to serve up (for example) web pages at a rate where the downloaded data far exceeds the stored data or any other use case where a small amount of data is served up a large amount of times This is the part that ruled it out for me
    - eropple 9y ago
      Having trouble with hot objects definitely rules it out for me. Thanks!
deleted 9y ago
[deleted]
pinewurst 9y ago
I can't speak for the quality of other object storage providers, but being in the storage business I can say that if someone is running Ceph, find another provider.
- kawera 9y ago
  Care to elaborate a little?
- scurvy 9y ago
  This is pure conjecture and FUD. Lemme guess, you work at EMC? I can refute your comment by saying that we have never lost a bit of data while running Ceph. I can't say the same for EMC tech (Isilon).
  - nik736 9y ago
    A lot of companies using Ceph at scale are facing huge issues (OVH, etc.), so he is not wrong. Why take the risk of going with a solution that is known to cause issues? Has nothing to do with EMC.
    - true_religion 9y ago
      I believe OVH uses openstack and not ceph.
      - nik736 9y ago
        You can use Ceph together with OpenStack. They used ceph for their cloud services but had huge problems. If I am not mistaken they completely threw out Ceph by now. http://travaux.ovh.net/?do=details&id=20636 http://travaux.ovh.net/?do=details&id=20636 http://status.ovh.net/?do=details&id=14139 http://status.ovh.net/?do=details&id=14139 http://travaux.ovh.com/?do=details&id=20490 http://travaux.ovh.com/?do=details&id=20490 http://travaux.ovh.net/?do=details&id=26382 http://travaux.ovh.net/?do=details&id=26382 and so on... They took months to figure out what's going on.
Operyl 9y ago
I mean, Digital Ocean itself has been “jokeish”. I had a VM ago down for 12 hours, it took 6 hours to even get them to confirm they had an issue on that machine. It was stupid.
doh 9y ago
I can't speak about S3 but we're using heavily GCS (google cloud storage) and haven't experienced any problems. From time to time we can see a slowdown, but never seen an outage. I also looked at B2 [0] once or twice. The price is great, but the traffic cost (egress from GCE) renders it unusable for us. [0] https://www.backblaze.com/b2/cloud-storage.html https://www.backblaze.com/b2/cloud-storage.html
- eropple 9y ago
  $0.12/GB egress for GCS is even higher than S3, though. :/
  - mattbillenstein 9y ago
    The bandwidth charges are hefty if you're moving a lot of bits, but I wouldn't use anything other than S3 or GCS probably -- the other guys just don't have a track record of reliability yet. But, you can build a poor-man's CDN -- varnish caches on DO/Linode/whatever where you get multiple terabytes of bandwidth for a small VM. So, you use the best object storage provider, but move most of the bits cheaply using Varnish + Route53 geo-dns.
    - eropple 9y ago
      That's definitely on the table. Time-consuming, though.
      - mattbillenstein 9y ago
        It's more to manage for sure, but something that can be built in a day or three.
        avtar 9y ago
        Building is arguably the easy part. Managing and supporting is a different story.
zedpm 9y ago
>S3's pricing is absolutely ridiculous by comparison, but they have the reliability argument on their side... Well, unless your S3 buckets are in us-east-1. For some reason Amazon keeps having issues with S3 in that region. Since the storage costs appear to be the same between Spaces and S3 ($0.02/GB/month) and neither charge for inbound transfer, I'm assuming your problem is with the outbound transfer pricing (S3 charges 9x what DO charges) and/or the per-request pricing. GCP's Regional Cloud Storage has the same storage pricing, even higher outbound transfer pricing, and the same request pricing. I haven't looked at any other providers, but if you want reliability, you're going to have to pay for it.
- eropple 9y ago
  It is external egress pricing, yeah. And CloudFront doesn't really help (it's still pricey). As this is mostly a side project, I think I can live with being a little adventurous.
- tedivm 9y ago
  Their us-east-1 region is notorious for being the region with the least reliability. Unless there's a good reason not to I always recommend people default to us-west-2.
squid3 9y ago
You may find NodeChef's object storage the right solution. https://nodechef.com/s3-compatible-object-storage https://nodechef.com/s3-compatible-object-storage
- eropple 9y ago
  Thanks for the suggestion, but a dollar per gigabyte per month is ridiculous unless you're downloading everything in your store eleven times a month. Even S3 only costs $0.02/GB storage and $0.09/GB egress.
acd 9y ago
There is Sheepdog distributed file system which recently went into version 1.0. Sheepdog is similar to Ceph but seem simpler to operate. For distributed object storage: I have also used MooseFS,LizardFS distributed object storage and MooseFS,Lizard runs very steady on production work loads. Steady as setup and then no ops issues. Also to the short list is BeeGFS, BeeGFS is created by Fraunhofer is seriously fast distributed file system.
halbritt 9y ago
Minio is cool. Unfortunately, performance is anyone's guess. It has erasure coding as well. You could deploy on bare VM's with local storage in any cloud provider and have no dependency on network blocked storage. With k8s 1.10 you get persistent local storage as well, as such you could probably build a fairly highly available system. Pro tip: do it in GCP as they have nice local SSDs you can attach to any instance. They're 375GB, 25k IOPS, and $.08GB, way cheaper than AWS I2 instances.
- eropple 9y ago
  So I have no problem with network storage. I have a cost problem (the project I'm working on is not intended to make a bunch of money and I'm trying to keep costs low so I can keep prices extremely low). What you're describing would functionally be even more expensive than just using S3 directly, if it were to be done in AWS or in GCS. Right now the leading (uncomfortable) solution is probably DigitalOcean Spaces and a little bit of prayer.
fapjacks 9y ago
This isn't related to block storage at all, but I was a big fan of DO until I hit a weird issue where they wanted me to prepay via PayPal to spin up more than 50 droplets at a time. I work in an organization that is spinning up many nodes at once for a short time and then destroying them soon after, for various but totally legitimate reasons. One look at our account history can demonstrate that this is almost exclusively how we use their services, so it's not like this was a weird request. And we've never missed a payment or paid late or otherwise ever given Digital Ocean any reason to think we wouldn't be good for the charges at the end of the month (especially considering we were already spending sometimes in the thousands of dollars every month). This was so off-putting. I stopped using Digital Ocean that day.
- mike_j 9y ago
  You could’ve asked to have been moved to a business account. Just saying.
  - fapjacks 9y ago
    We do have a business account, if I'm reading the console correctly. I was wrong about the limit of 50. It is a limit of 100 droplets but this is an artificial limitation that they wouldn't budge on. It's clear from the (lengthy) account history that this is normal behavior for us, and I work for a company whose name everyone knows, so it's not like we're some no-name scammers. Regardless, asking customers to buy vouchers for what really is only moderate use of a service is really off-putting. It was so off-putting actually that, more than halfway through writing it, I scrapped a driver for some pretty popular software that would have enabled the use of Digital Ocean as a backend.
    - cube2222 9y ago
      Wasn't the issue resolved after writing to their support?
      - fapjacks 9y ago
onestone 9y ago
StorPool[0]. Inoreader recently blogged about their experience with it, and why they chose it instead of Ceph[1]. [0] https://storpool.com/ https://storpool.com/ [1] http://blog.inoreader.com/2018/03/success-story-inoreader-opennebula-storpool.html http://blog.inoreader.com/2018/03/success-story-inoreader-op...
- eropple 9y ago
  But...I don't care about the technology. I care about the object storage available to me without caring about the technology. So what's this do for me?

aedocw 9y ago

I would suggest you take a look at Wasabi for object storage. I'm just a customer, but have been using them for close to a year for off-site backup storage and it's been great.

filleokus 9y ago
Reposting from my comment from yesterday[0], how are the speeds from you location (and where's that)? It have been ridiculously slow from Northern Europe when I've tried it, like not even 1 MB/s down. 0: https://news.ycombinator.com/item?id=16726324 https://news.ycombinator.com/item?id=16726324
eropple 9y ago
Wasabi looks interesting. What's the catch? How's the outgoing bandwidth/download speed? (Large file downloads are my main concern.)
- corobo 9y ago
  > Wasabi’s hot cloud storage service is not designed to be used to serve up (for example) web pages at a rate where the downloaded data far exceeds the stored data or any other use case where a small amount of data is served up a large amount of times This is the part that ruled it out for me
  - eropple 9y ago
    Having trouble with hot objects definitely rules it out for me. Thanks!

deleted 9y ago

[deleted]

pinewurst 9y ago

I can't speak for the quality of other object storage providers, but being in the storage business I can say that if someone is running Ceph, find another provider.

kawera 9y ago
Care to elaborate a little?
scurvy 9y ago
This is pure conjecture and FUD. Lemme guess, you work at EMC? I can refute your comment by saying that we have never lost a bit of data while running Ceph. I can't say the same for EMC tech (Isilon).
- nik736 9y ago
  A lot of companies using Ceph at scale are facing huge issues (OVH, etc.), so he is not wrong. Why take the risk of going with a solution that is known to cause issues? Has nothing to do with EMC.
  - true_religion 9y ago
    I believe OVH uses openstack and not ceph.
    - nik736 9y ago
      You can use Ceph together with OpenStack. They used ceph for their cloud services but had huge problems. If I am not mistaken they completely threw out Ceph by now. http://travaux.ovh.net/?do=details&id=20636 http://travaux.ovh.net/?do=details&id=20636 http://status.ovh.net/?do=details&id=14139 http://status.ovh.net/?do=details&id=14139 http://travaux.ovh.com/?do=details&id=20490 http://travaux.ovh.com/?do=details&id=20490 http://travaux.ovh.net/?do=details&id=26382 http://travaux.ovh.net/?do=details&id=26382 and so on... They took months to figure out what's going on.

Operyl 9y ago

I mean, Digital Ocean itself has been “jokeish”. I had a VM ago down for 12 hours, it took 6 hours to even get them to confirm they had an issue on that machine. It was stupid.

doh 9y ago

I can't speak about S3 but we're using heavily GCS (google cloud storage) and haven't experienced any problems. From time to time we can see a slowdown, but never seen an outage. I also looked at B2 [0] once or twice. The price is great, but the traffic cost (egress from GCE) renders it unusable for us. [0] https://www.backblaze.com/b2/cloud-storage.html https://www.backblaze.com/b2/cloud-storage.html

eropple 9y ago
$0.12/GB egress for GCS is even higher than S3, though. :/
- mattbillenstein 9y ago
  The bandwidth charges are hefty if you're moving a lot of bits, but I wouldn't use anything other than S3 or GCS probably -- the other guys just don't have a track record of reliability yet. But, you can build a poor-man's CDN -- varnish caches on DO/Linode/whatever where you get multiple terabytes of bandwidth for a small VM. So, you use the best object storage provider, but move most of the bits cheaply using Varnish + Route53 geo-dns.
  - eropple 9y ago
    That's definitely on the table. Time-consuming, though.
    - mattbillenstein 9y ago
      It's more to manage for sure, but something that can be built in a day or three.
      - avtar 9y ago
        Building is arguably the easy part. Managing and supporting is a different story.

zedpm 9y ago

>S3's pricing is absolutely ridiculous by comparison, but they have the reliability argument on their side... Well, unless your S3 buckets are in us-east-1. For some reason Amazon keeps having issues with S3 in that region. Since the storage costs appear to be the same between Spaces and S3 ($0.02/GB/month) and neither charge for inbound transfer, I'm assuming your problem is with the outbound transfer pricing (S3 charges 9x what DO charges) and/or the per-request pricing. GCP's Regional Cloud Storage has the same storage pricing, even higher outbound transfer pricing, and the same request pricing. I haven't looked at any other providers, but if you want reliability, you're going to have to pay for it.

eropple 9y ago
It is external egress pricing, yeah. And CloudFront doesn't really help (it's still pricey). As this is mostly a side project, I think I can live with being a little adventurous.
tedivm 9y ago
Their us-east-1 region is notorious for being the region with the least reliability. Unless there's a good reason not to I always recommend people default to us-west-2.

squid3 9y ago

You may find NodeChef's object storage the right solution. https://nodechef.com/s3-compatible-object-storage https://nodechef.com/s3-compatible-object-storage

eropple 9y ago
Thanks for the suggestion, but a dollar per gigabyte per month is ridiculous unless you're downloading everything in your store eleven times a month. Even S3 only costs $0.02/GB storage and $0.09/GB egress.

acd 9y ago

There is Sheepdog distributed file system which recently went into version 1.0. Sheepdog is similar to Ceph but seem simpler to operate. For distributed object storage: I have also used MooseFS,LizardFS distributed object storage and MooseFS,Lizard runs very steady on production work loads. Steady as setup and then no ops issues. Also to the short list is BeeGFS, BeeGFS is created by Fraunhofer is seriously fast distributed file system.

halbritt 9y ago

Minio is cool. Unfortunately, performance is anyone's guess. It has erasure coding as well. You could deploy on bare VM's with local storage in any cloud provider and have no dependency on network blocked storage. With k8s 1.10 you get persistent local storage as well, as such you could probably build a fairly highly available system. Pro tip: do it in GCP as they have nice local SSDs you can attach to any instance. They're 375GB, 25k IOPS, and $.08GB, way cheaper than AWS I2 instances.

eropple 9y ago
So I have no problem with network storage. I have a cost problem (the project I'm working on is not intended to make a bunch of money and I'm trying to keep costs low so I can keep prices extremely low). What you're describing would functionally be even more expensive than just using S3 directly, if it were to be done in AWS or in GCS. Right now the leading (uncomfortable) solution is probably DigitalOcean Spaces and a little bit of prayer.

fapjacks 9y ago

This isn't related to block storage at all, but I was a big fan of DO until I hit a weird issue where they wanted me to prepay via PayPal to spin up more than 50 droplets at a time. I work in an organization that is spinning up many nodes at once for a short time and then destroying them soon after, for various but totally legitimate reasons. One look at our account history can demonstrate that this is almost exclusively how we use their services, so it's not like this was a weird request. And we've never missed a payment or paid late or otherwise ever given Digital Ocean any reason to think we wouldn't be good for the charges at the end of the month (especially considering we were already spending sometimes in the thousands of dollars every month). This was so off-putting. I stopped using Digital Ocean that day.

mike_j 9y ago
You could’ve asked to have been moved to a business account. Just saying.
- fapjacks 9y ago
  We do have a business account, if I'm reading the console correctly. I was wrong about the limit of 50. It is a limit of 100 droplets but this is an artificial limitation that they wouldn't budge on. It's clear from the (lengthy) account history that this is normal behavior for us, and I work for a company whose name everyone knows, so it's not like we're some no-name scammers. Regardless, asking customers to buy vouchers for what really is only moderate use of a service is really off-putting. It was so off-putting actually that, more than halfway through writing it, I scrapped a driver for some pretty popular software that would have enabled the use of Digital Ocean as a backend.
  - cube2222 9y ago
    Wasn't the issue resolved after writing to their support?
    - fapjacks 9y ago

onestone 9y ago

StorPool[0]. Inoreader recently blogged about their experience with it, and why they chose it instead of Ceph[1]. [0] https://storpool.com/ https://storpool.com/ [1] http://blog.inoreader.com/2018/03/success-story-inoreader-opennebula-storpool.html http://blog.inoreader.com/2018/03/success-story-inoreader-op...

eropple 9y ago
But...I don't care about the technology. I care about the object storage available to me without caring about the technology. So what's this do for me?