3 ms·

Limited Availability of Spaces API in NYC3

28 points by turrini 8y ago

hemancuso 8y ago
What is the replication factor on these objects? 0.1% objects unavailable is a worrisome number as it suggests a meaningful number of objects are under-replicated and the cluster is poorly configured. I hope DO speaks to the durability of Spaces in a meaningful fashion after this is sorted out. Multiple days of having a large amount of data unavailable should shake the confidence of anyone considering Spaces for object storage. Are objects stored with 3 replicas? Are they on heavily striped volumes? What is going on that could cause this failure mode and what's stopping a much larger failure? Lastly, nowhere in their status updates are they saying "don't worry, no data-loss, just unavailable for a bit" - which I read as "we hope to not lose any data, but no promises"
Operyl 8y ago
This is why I have consistently seen people stay clear of Digital Ocean. They’re great for pet projects, and staging/development environments (as was their original intention when they came out if I recall correctly) but many of these “small” issues can quickly become a huge headache. Sure, anybody with an ounce of sanity has a backup plan, but the reality is none of us want to have to jump into it unless absolutely required. On a side note: you should test your backups and plans often! Don’t let outages catch you off guard.
- holdenc 8y ago
  I'm a customer at linode, vultr, and digital ocean, and digital ocean (over the last 6 years in their NYC datacenter) has had exceptional uptime and physical host integrity. I've never had cycles stolen by "noisy neighbors" (as has happened at Linode) or lost a server (as is to be expected on EC2). Their unixbench scores are great (fast disks and CPUs). All that said -- you want use a new product, expect to pay the pioneer tax. Their CDN is even greener (no built-in GZIP, broken CORs header forwarding, only pulls from DO spaces, no CNAME mapping) but the bandwidth is free/dirt cheap. They are very upfront about their offering, and don't seem to be over promising, or over selling. You get what you pay for, and I am ok with that.
  - Operyl 8y ago
    I guess my problem is they’re not ultimately some new product. Based on previous outages for this same product they’re basically reselling Ceph. There’s no new technology here.
- mlthoughts2018 8y ago
  I had the chance to do significant interviews with DO for a senior position, and it made me so skeptical. They seem more interested in the shortest path to an inflated IPO than in basic investment in their product offerings. They also seemed overly bureaucratic for a company of their size / type. A lot of decision making and investment on the tech side seemed to roll up to a particular business operations executive who talked about buzzwords and seemed like he thinks TED talks are reality. In other parts of the interview, people explained to me how there was some prescriptive slide deck with 30+ slides on all the minutiae of how Agile would work for a certain team. Brainfucked with process and not informed by engineering experts. To top all that off, their compensation offer was way below market, no bonus, and very poor equity terms, all after telling me they were very excited to hire me. It really seemed like a bleak place.
alphabettsy 8y ago
I transferred a 1TB Cryptomator vault to Spaces to test it and I’m finding quite a few files unreadable with errors. This is not an issue I’ve had with any other storage provider. They’re easily reproducible files, but I’ve lost dozens of files at this point, which is unacceptable for a storage offering.

28 points by turrini 8y ago

What is the replication factor on these objects? 0.1% objects unavailable is a worrisome number as it suggests a meaningful number of objects are under-replicated and the cluster is poorly configured. I hope DO speaks to the durability of Spaces in a meaningful fashion after this is sorted out. Multiple days of having a large amount of data unavailable should shake the confidence of anyone considering Spaces for object storage. Are objects stored with 3 replicas? Are they on heavily striped volumes? What is going on that could cause this failure mode and what's stopping a much larger failure? Lastly, nowhere in their status updates are they saying "don't worry, no data-loss, just unavailable for a bit" - which I read as "we hope to not lose any data, but no promises"

Operyl 8y ago

This is why I have consistently seen people stay clear of Digital Ocean. They’re great for pet projects, and staging/development environments (as was their original intention when they came out if I recall correctly) but many of these “small” issues can quickly become a huge headache. Sure, anybody with an ounce of sanity has a backup plan, but the reality is none of us want to have to jump into it unless absolutely required. On a side note: you should test your backups and plans often! Don’t let outages catch you off guard.

holdenc 8y ago
I'm a customer at linode, vultr, and digital ocean, and digital ocean (over the last 6 years in their NYC datacenter) has had exceptional uptime and physical host integrity. I've never had cycles stolen by "noisy neighbors" (as has happened at Linode) or lost a server (as is to be expected on EC2). Their unixbench scores are great (fast disks and CPUs). All that said -- you want use a new product, expect to pay the pioneer tax. Their CDN is even greener (no built-in GZIP, broken CORs header forwarding, only pulls from DO spaces, no CNAME mapping) but the bandwidth is free/dirt cheap. They are very upfront about their offering, and don't seem to be over promising, or over selling. You get what you pay for, and I am ok with that.
- Operyl 8y ago
  I guess my problem is they’re not ultimately some new product. Based on previous outages for this same product they’re basically reselling Ceph. There’s no new technology here.
mlthoughts2018 8y ago
I had the chance to do significant interviews with DO for a senior position, and it made me so skeptical. They seem more interested in the shortest path to an inflated IPO than in basic investment in their product offerings. They also seemed overly bureaucratic for a company of their size / type. A lot of decision making and investment on the tech side seemed to roll up to a particular business operations executive who talked about buzzwords and seemed like he thinks TED talks are reality. In other parts of the interview, people explained to me how there was some prescriptive slide deck with 30+ slides on all the minutiae of how Agile would work for a certain team. Brainfucked with process and not informed by engineering experts. To top all that off, their compensation offer was way below market, no bonus, and very poor equity terms, all after telling me they were very excited to hire me. It really seemed like a bleak place.

alphabettsy 8y ago

I transferred a 1TB Cryptomator vault to Spaces to test it and I’m finding quite a few files unreadable with errors. This is not an issue I’ve had with any other storage provider. They’re easily reproducible files, but I’ve lost dozens of files at this point, which is unacceptable for a storage offering.