16 ms·
FRA1 Block Storage Issue
- eropple 9y agoAs somebody who's been looking really hard at a project/side business that'd use Spaces (DO's object storage system), this makes me super, super nervous. To say nothing of block storage--yikes. Can anyone speak to quality/reliability of other object storage providers that have S3-compatible (including presigned URL) APIs? S3's pricing is absolutely ridiculous by comparison, but they have the reliability argument on their side...
- aedocw 9y agoI would suggest you take a look at Wasabi for object storage. I'm just a customer, but have been using them for close to a year for off-site backup storage and it's been great.
- filleokus 9y agoReposting from my comment from yesterday[0], how are the speeds from you location (and where's that)? It have been ridiculously slow from Northern Europe when I've tried it, like not even 1 MB/s down. 0: https://news.ycombinator.com/item?id=16726324 https://news.ycombinator.com/item?id=16726324
- eropple 9y agoWasabi looks interesting. What's the catch? How's the outgoing bandwidth/download speed? (Large file downloads are my main concern.)
- corobo 9y ago> Wasabi’s hot cloud storage service is not designed to be used to serve up (for example) web pages at a rate where the downloaded data far exceeds the stored data or any other use case where a small amount of data is served up a large amount of times This is the part that ruled it out for me
- eropple 9y agoHaving trouble with hot objects definitely rules it out for me. Thanks!
- deleted 9y ago[deleted]
- pinewurst 9y agoI can't speak for the quality of other object storage providers, but being in the storage business I can say that if someone is running Ceph, find another provider.
- kawera 9y agoCare to elaborate a little?
- scurvy 9y agoThis is pure conjecture and FUD. Lemme guess, you work at EMC? I can refute your comment by saying that we have never lost a bit of data while running Ceph. I can't say the same for EMC tech (Isilon).
- nik736 9y agoA lot of companies using Ceph at scale are facing huge issues (OVH, etc.), so he is not wrong. Why take the risk of going with a solution that is known to cause issues? Has nothing to do with EMC.
- true_religion 9y agoI believe OVH uses openstack and not ceph.
- nik736 9y agoYou can use Ceph together with OpenStack. They used ceph for their cloud services but had huge problems. If I am not mistaken they completely threw out Ceph by now. http://travaux.ovh.net/?do=details&id=20636 http://travaux.ovh.net/?do=details&id=20636 http://status.ovh.net/?do=details&id=14139 http://status.ovh.net/?do=details&id=14139 http://travaux.ovh.com/?do=details&id=20490 http://travaux.ovh.com/?do=details&id=20490 http://travaux.ovh.net/?do=details&id=26382 http://travaux.ovh.net/?do=details&id=26382 and so on... They took months to figure out what's going on.
- Operyl 9y agoI mean, Digital Ocean itself has been “jokeish”. I had a VM ago down for 12 hours, it took 6 hours to even get them to confirm they had an issue on that machine. It was stupid.
- doh 9y agoI can't speak about S3 but we're using heavily GCS (google cloud storage) and haven't experienced any problems. From time to time we can see a slowdown, but never seen an outage. I also looked at B2 [0] once or twice. The price is great, but the traffic cost (egress from GCE) renders it unusable for us. [0] https://www.backblaze.com/b2/cloud-storage.html https://www.backblaze.com/b2/cloud-storage.html
- eropple 9y ago$0.12/GB egress for GCS is even higher than S3, though. :/
- mattbillenstein 9y agoThe bandwidth charges are hefty if you're moving a lot of bits, but I wouldn't use anything other than S3 or GCS probably -- the other guys just don't have a track record of reliability yet. But, you can build a poor-man's CDN -- varnish caches on DO/Linode/whatever where you get multiple terabytes of bandwidth for a small VM. So, you use the best object storage provider, but move most of the bits cheaply using Varnish + Route53 geo-dns.
- eropple 9y agoThat's definitely on the table. Time-consuming, though.
- mattbillenstein 9y agoIt's more to manage for sure, but something that can be built in a day or three.
- avtar 9y agoBuilding is arguably the easy part. Managing and supporting is a different story.
- zedpm 9y ago>S3's pricing is absolutely ridiculous by comparison, but they have the reliability argument on their side... Well, unless your S3 buckets are in us-east-1. For some reason Amazon keeps having issues with S3 in that region. Since the storage costs appear to be the same between Spaces and S3 ($0.02/GB/month) and neither charge for inbound transfer, I'm assuming your problem is with the outbound transfer pricing (S3 charges 9x what DO charges) and/or the per-request pricing. GCP's Regional Cloud Storage has the same storage pricing, even higher outbound transfer pricing, and the same request pricing. I haven't looked at any other providers, but if you want reliability, you're going to have to pay for it.
- eropple 9y agoIt is external egress pricing, yeah. And CloudFront doesn't really help (it's still pricey). As this is mostly a side project, I think I can live with being a little adventurous.
- tedivm 9y agoTheir us-east-1 region is notorious for being the region with the least reliability. Unless there's a good reason not to I always recommend people default to us-west-2.
- squid3 9y agoYou may find NodeChef's object storage the right solution. https://nodechef.com/s3-compatible-object-storage https://nodechef.com/s3-compatible-object-storage
- eropple 9y agoThanks for the suggestion, but a dollar per gigabyte per month is ridiculous unless you're downloading everything in your store eleven times a month. Even S3 only costs $0.02/GB storage and $0.09/GB egress.
- acd 9y agoThere is Sheepdog distributed file system which recently went into version 1.0. Sheepdog is similar to Ceph but seem simpler to operate. For distributed object storage: I have also used MooseFS,LizardFS distributed object storage and MooseFS,Lizard runs very steady on production work loads. Steady as setup and then no ops issues. Also to the short list is BeeGFS, BeeGFS is created by Fraunhofer is seriously fast distributed file system.
- halbritt 9y agoMinio is cool. Unfortunately, performance is anyone's guess. It has erasure coding as well. You could deploy on bare VM's with local storage in any cloud provider and have no dependency on network blocked storage. With k8s 1.10 you get persistent local storage as well, as such you could probably build a fairly highly available system. Pro tip: do it in GCP as they have nice local SSDs you can attach to any instance. They're 375GB, 25k IOPS, and $.08GB, way cheaper than AWS I2 instances.
- eropple 9y agoSo I have no problem with network storage. I have a cost problem (the project I'm working on is not intended to make a bunch of money and I'm trying to keep costs low so I can keep prices extremely low). What you're describing would functionally be even more expensive than just using S3 directly, if it were to be done in AWS or in GCS. Right now the leading (uncomfortable) solution is probably DigitalOcean Spaces and a little bit of prayer.
- fapjacks 9y agoThis isn't related to block storage at all, but I was a big fan of DO until I hit a weird issue where they wanted me to prepay via PayPal to spin up more than 50 droplets at a time. I work in an organization that is spinning up many nodes at once for a short time and then destroying them soon after, for various but totally legitimate reasons. One look at our account history can demonstrate that this is almost exclusively how we use their services, so it's not like this was a weird request. And we've never missed a payment or paid late or otherwise ever given Digital Ocean any reason to think we wouldn't be good for the charges at the end of the month (especially considering we were already spending sometimes in the thousands of dollars every month). This was so off-putting. I stopped using Digital Ocean that day.
- mike_j 9y agoYou could’ve asked to have been moved to a business account. Just saying.
- fapjacks 9y agoWe do have a business account, if I'm reading the console correctly. I was wrong about the limit of 50. It is a limit of 100 droplets but this is an artificial limitation that they wouldn't budge on. It's clear from the (lengthy) account history that this is normal behavior for us, and I work for a company whose name everyone knows, so it's not like we're some no-name scammers. Regardless, asking customers to buy vouchers for what really is only moderate use of a service is really off-putting. It was so off-putting actually that, more than halfway through writing it, I scrapped a driver for some pretty popular software that would have enabled the use of Digital Ocean as a backend.
- onestone 9y agoStorPool[0]. Inoreader recently blogged about their experience with it, and why they chose it instead of Ceph[1]. [0] https://storpool.com/ https://storpool.com/ [1] http://blog.inoreader.com/2018/03/success-story-inoreader-opennebula-storpool.html http://blog.inoreader.com/2018/03/success-story-inoreader-op...
- eropple 9y agoBut...I don't care about the technology. I care about the object storage available to me without caring about the technology. So what's this do for me?
- qertoip 9y agoCan confirm. We run 26 services in production on DigitalOcean. Every single VPS in our setup uses block storage feature as a persistence layer (system logs, app logs, databases, etc). Thanks to this architecture we can rebuild machines at will. The _function_ and _state_ are nicely separated. The downside is that we are now fucked.
- mattbillenstein 9y agoYou should re-architect without network block storage as a requirement imho. Ever since that big AWS EBS outage in 2012 or whatever, I avoid it like the plague. Databases like low-latency local storage (the newer nvme instances on aws are very good) and logs and whatnot are aggregated with other systems (fluent, logstash, etc). I actually do not miss EBS much at all really -- if I have a problem with a VM, they're disposable and redundant and I've designed out most of the SPOFs.
- qertoip 9y agoWe are actually very happy with database performance over block storage. As for the architecture, in DigitalOcean the external block storage is the only way to separate function from state. We need this to be able to rebuild VPS-es regularly. This is to avoid configuration drift and to prove/enforce reproducibility.
- mattbillenstein 9y agoI guess I've solved that in other ways -- configuration drift is really a config management and orchestration problem. And recycling or OS upgrades can happen if your story around replication is sound -- I only typically do that every two years anyway following the Ubuntu LTS releases.
- joelhaasnoot 9y agoComing from an enterprise background for a couple of years: network block storage screws you one way or another either way. It's a single point of failure that no matter how reliable it is, will always go down at some point or have failures that affect your application layer.
- pstrateman 9y ago100% of the downtime for MomentoVPS was from ceph cluster failures....
- teilo 9y agoI have heard of a number of Ceph nightmares like this. A few years back, Logos Bible Software, who has a Ceph-based content platform (a huge library of e-books with a massive amount of meta-data), was down for a week because of a cascading Ceph cluster failure. It really doesn't speak well of the Ceph architecture. It is highly performant, but at what cost? Failures on this scale can ruin a business.
- deleted 9y ago[deleted]
- zzzcpan 9y agoWell, you can always partition a large cluster into many small clusters and prevent cascading failures, other issues from affecting everyone or getting too long to recover from. This is like a very basic reliability technique everyone should know.
- unilynx 9y agoThey've just started sending out SLA credit notices: ----- Hello, On 2018-04-01 at 7:08 UTC, one of several storage clusters in our FRA1 region suffered a cascading failure. As a result, multiple redundant hosts in the storage cluster suffered an Out Of Memory (OOM) condition and crashed nearly simultaneously. We have identified that you, or your team account, were impacted by this incident and will grant an SLA credit equal to 30% of your entire Block Storage spend for April, not just usage in FRA1. This credit will appear on your account at the end of April, and will be reflected on your April 2018 invoice. We apologize for the incident and recognize the impact this outage had on your work and business. You can read the full detail of our public post-mortem here: http://status.digitalocean.com/incidents/8sk3mbgp6jgl http://status.digitalocean.com/incidents/8sk3mbgp6jgl Thank you, Team DigitalOcean