4 ms·

BunnyCDN has been silently losing our production files for 15 months

180 points by speckx 7mo ago

s09dfhks 7mo ago
Interesting timing given the post yesterday about someone switching to BunnyCDN from cloudflare
- Havoc 7mo ago
  To be fair the Reddit user account is 8 years old. Not conclusive but does suggest it may be organic
  - sergiotapia 7mo ago
    you can purchase reddit accounts with age/content.
- DANmode 7mo ago
  That is interesting.
nh43215rgb 7mo ago
A time for scrutiny...
reddalo 7mo ago
That's scary. I'm in the process of moving all of my services to Europe, and I had considered BunnyCDN, but after this I'm not so sure anymore. I also tried Hetzner Object Storage. I love Hetzner, they're great, except for their Object Storage service, which is completely unreliable (errors, slowness, etc.). I'm surprised that Hetzner still hasn't retired that product until it's properly fixed. My last chance is with Scaleway. Xavier Niel's products are always good, so fingers crossed...
- 12907835202 7mo ago
  I guess you may have alot of files, but to me object storage is so cheap I keep copies on Aws S3, wasabi, r2 and a 16 TB HDD on my hetzner server. Admittedly 4 might be too many. But at some point I switched to r2 for the free public egress and deleting one of Aws or wasabi has never been a priority and I don't want to do it without putting in the time to quadruple check I'm not deleting anything important. But at the very least id have 3. I'd hate to discover that my S3 was blown up in a war and then copying everything my HDD was the last straw that pushed an aging drive over the edge.
- pier25 7mo ago
  Bunny's CDN is great. The issue in the post is about their file storage service not the CDN. For my storage needs in the EU I'm using Upcloud's object storage. Very happy with them. Then a Bunny cdn zone if I need to share the files publicly.
- champtar 7mo ago
  There is also OVH (not affiliated, just happily using their VPS), but I would consider switching to 2 providers, as any provider can have data loss or just lock your account at anytime for any reasons. 2 providers with free egress so you can easily replicate data between them.
  - WhyNotHugo 7mo ago
    The classic rule is 3-2-1: - Three copies - Two types of media - One off-site or offline
- miyuru 7mo ago
  >My last chance is with Scaleway. I have been using for sw for object storage for a while. I haven't had a a issue with them with object storage. only problem I had was with a another product called cockpit(they ship metrics about object storage there), which they bundled with object storage product, which cannot be disabled. This month I had a 12 euro surcharge, because they enabled something in cockpit by default a while ago and suddenly start to charging for it.
- 7mo ago
carverauto 7mo ago
[flagged]
m3nu 7mo ago
Also a user of their CDN, storage and container service for about 4 years. My experience was more positive. One time I had a minor issue with their storage where I couldn't replace a file or something. This was fairly early after the product launched. They fixed it and gave me free credit for reporting it.
- DANmode 7mo ago
  If the story is true, remaining a user seems…terrifying?
  - ipaddr 7mo ago
    Because they fixed a minor issue many years ago?
    - zamadatix 7mo ago
      I believe "the story" was intended to refer to the horror story in the main post rather than the minor issue described in the comment above.
      - DANmode 7mo ago
        100%
kjs3 7mo ago
Wait...this has been going on for 15 months without resolution or recompense and you haven't pulled up stakes and moved to almost anyone else? I get "it's a lot of work to move" and maybe "we hope they'll figure this out so we don't have to move", but in my world that excuse runs out waaaaaay before 15 months. The people I'm accountable to would have hauled me out back and put me out of their misery after, maybe, 6 months on the outside.
- simoncion 7mo ago
  > ...but in my world that excuse runs out waaaaaay before 15 months. I expect the combination of Yes, we should've migrated away sooner, we never had the capacity to do so and hoped Bunny would just get their shit together. and The loss rate isn't enormous in percentage terms, but it's consistent and ongoing. means that detecting and dealing with the loss is substantially less work than moving away. [0] I expect that Management is fully aware of what's up and is making the call here. [0] "Just" add a retry if the post-upload verification step fails! Sure, it's slower, but it works, right??? mournful sob
- WhyNotHugo 7mo ago
  I'd guess that it hasn't become an issue with any of their clients. If any big client had been impacted once, they would have sped up moving. If a big client had been impacted twice, they would have moved already.
  - kjs3 7mo ago
    That's fair, and I agree (hope?) if it was customer impacting there would be a miraculous allocation of scarce resources to move platforms. But in my world I'd get asked "why are you still screwing around with a vendor that clearly sucks" even if it was for a non-prod playpen of no consequence. YMMV.
getcrunk 7mo ago
I’ve used bunny for a few years … happily. I wonder if this is a bug due to some meta data of the files like the names or something. Very weird. Good thing you had metrics to catch it. I upload all object storage stuff to bunny for live but also to backblace for backup. I’ve always wanted to implement fail over client side for any asset over to bacblaze but seems like a lot of overhead
sergiotapia 7mo ago
I'm a very grateful bunnycdn customer. they are great, lovely UX and great performance and price. we use them to store our image files and other documents.
dgl 7mo ago
I tried using their Magic Containers product and there were issues that showed a lack of attention to detail as well. It's supposed to scale globally (magically!) but I found multiple cases where particular nodes were problematic and the health checks didn't detect them (in fact to start with the health checks didn't even work properly if you had multiple containers, they did fix that). The support was quite slow too, after finding multiple product issues they'd escalate to developers and then come back a month later and ask to retest, but some of this took multiple round trips. I was only using this on a side project, but definitely wouldn't consider them for anything critical, even if they are quite cheap.
- tayk47999 7mo ago
  [dead]
benatkin 7mo ago
On LowEndTalk: https://lowendtalk.com/discussion/comment/4767400/#Comment_4767400 https://lowendtalk.com/discussion/comment/4767400/#Comment_4...
Volundr 7mo ago
Reply in thread from Bunny: https://old.reddit.com/r/webdev/comments/1sglytg/bunnycdn_has_been_silently_losing_our_production/of7fogs/ https://old.reddit.com/r/webdev/comments/1sglytg/bunnycdn_ha...
- benatkin 7mo ago
  [flagged]
  - john_strinlai 7mo ago
    the full quote is: "does not matter if this is an isolated edge case. Data loss for 15 months with poor support isn't something that can be waved off as 'edge case'. The fact I had to go to Reddit to get someone's attention for this is insane. By the way, I tried reaching out to various 'senior management' personnel via LinkedIn last year and no-one replied. Escalation requests via the Support thread ignored and declined. Very professional of you guys." and that seems pretty damn reasonable of a reply. the fact they had to go to reddit is insane. a sarcastic "very professional of you guys" is pretty tame, given the situation, and not discrediting at all.
  - bakugo 7mo ago
    It's a perfectly understandable response to what is essentially just corporate damage control. I'd be pissed too if I was in their position. They've been ignoring the issue for over a year, yet it somehow took them only 4 hours after the reddit post was made to determine that it's "an isolated edge case", even though there's at least one other user reporting the same problem. The comment was clearly written with a focus on saving face rather than properly apologizing for the terrible support.
    - bobdvb 7mo ago
      Yeah. Someone recommended Bunny to me and I looked at it previously based on that but didn't have a use case then. But as someone who regularly recommends vendors for v.big CDN contracts I'm not impressed with their attempts to downplay things.
RobRivera 7mo ago
Handwaving of professionally agreed upon SLAs as an edgecase and providing poor customer support; thats a paddlin
faangguyindia 7mo ago
what's bunny? a CDN by Hug Hefner?
httpsterio 7mo ago
Not a great way to handle the issue originally, but I think the CEO's response is more than fair. I haven't had to contact Bunny's support so it's a bit disappointing to see this type of runaround from the support, but as an actual issue it seems pretty minor in the end. I've been using Bunny for ~four years after looking at video hosting. In the four years we've saved approximately 50 000€ compared to if we'd gone with Cloudflare video streaming service and it's just been rock solid for us.
- mort96 7mo ago
  The CEO's response is alright, the VP of engineering's response is a bit more worrying in how it mainly tries to downplay the issue.

180 points by speckx 7mo ago

s09dfhks 7mo ago

Interesting timing given the post yesterday about someone switching to BunnyCDN from cloudflare

Havoc 7mo ago
To be fair the Reddit user account is 8 years old. Not conclusive but does suggest it may be organic
- sergiotapia 7mo ago
  you can purchase reddit accounts with age/content.
DANmode 7mo ago
That is interesting.

nh43215rgb 7mo ago

A time for scrutiny...

reddalo 7mo ago

That's scary. I'm in the process of moving all of my services to Europe, and I had considered BunnyCDN, but after this I'm not so sure anymore. I also tried Hetzner Object Storage. I love Hetzner, they're great, except for their Object Storage service, which is completely unreliable (errors, slowness, etc.). I'm surprised that Hetzner still hasn't retired that product until it's properly fixed. My last chance is with Scaleway. Xavier Niel's products are always good, so fingers crossed...

12907835202 7mo ago
I guess you may have alot of files, but to me object storage is so cheap I keep copies on Aws S3, wasabi, r2 and a 16 TB HDD on my hetzner server. Admittedly 4 might be too many. But at some point I switched to r2 for the free public egress and deleting one of Aws or wasabi has never been a priority and I don't want to do it without putting in the time to quadruple check I'm not deleting anything important. But at the very least id have 3. I'd hate to discover that my S3 was blown up in a war and then copying everything my HDD was the last straw that pushed an aging drive over the edge.
pier25 7mo ago
Bunny's CDN is great. The issue in the post is about their file storage service not the CDN. For my storage needs in the EU I'm using Upcloud's object storage. Very happy with them. Then a Bunny cdn zone if I need to share the files publicly.
champtar 7mo ago
There is also OVH (not affiliated, just happily using their VPS), but I would consider switching to 2 providers, as any provider can have data loss or just lock your account at anytime for any reasons. 2 providers with free egress so you can easily replicate data between them.
- WhyNotHugo 7mo ago
  The classic rule is 3-2-1: - Three copies - Two types of media - One off-site or offline
miyuru 7mo ago
>My last chance is with Scaleway. I have been using for sw for object storage for a while. I haven't had a a issue with them with object storage. only problem I had was with a another product called cockpit(they ship metrics about object storage there), which they bundled with object storage product, which cannot be disabled. This month I had a 12 euro surcharge, because they enabled something in cockpit by default a while ago and suddenly start to charging for it.
7mo ago

carverauto 7mo ago

[flagged]

m3nu 7mo ago

Also a user of their CDN, storage and container service for about 4 years. My experience was more positive. One time I had a minor issue with their storage where I couldn't replace a file or something. This was fairly early after the product launched. They fixed it and gave me free credit for reporting it.

DANmode 7mo ago
If the story is true, remaining a user seems…terrifying?
- ipaddr 7mo ago
  Because they fixed a minor issue many years ago?
  - zamadatix 7mo ago
    I believe "the story" was intended to refer to the horror story in the main post rather than the minor issue described in the comment above.
    - DANmode 7mo ago
      100%

kjs3 7mo ago

Wait...this has been going on for 15 months without resolution or recompense and you haven't pulled up stakes and moved to almost anyone else? I get "it's a lot of work to move" and maybe "we hope they'll figure this out so we don't have to move", but in my world that excuse runs out waaaaaay before 15 months. The people I'm accountable to would have hauled me out back and put me out of their misery after, maybe, 6 months on the outside.

simoncion 7mo ago
> ...but in my world that excuse runs out waaaaaay before 15 months. I expect the combination of Yes, we should've migrated away sooner, we never had the capacity to do so and hoped Bunny would just get their shit together. and The loss rate isn't enormous in percentage terms, but it's consistent and ongoing. means that detecting and dealing with the loss is substantially less work than moving away. [0] I expect that Management is fully aware of what's up and is making the call here. [0] "Just" add a retry if the post-upload verification step fails! Sure, it's slower, but it works, right??? mournful sob
WhyNotHugo 7mo ago
I'd guess that it hasn't become an issue with any of their clients. If any big client had been impacted once, they would have sped up moving. If a big client had been impacted twice, they would have moved already.
- kjs3 7mo ago
  That's fair, and I agree (hope?) if it was customer impacting there would be a miraculous allocation of scarce resources to move platforms. But in my world I'd get asked "why are you still screwing around with a vendor that clearly sucks" even if it was for a non-prod playpen of no consequence. YMMV.

getcrunk 7mo ago

I’ve used bunny for a few years … happily. I wonder if this is a bug due to some meta data of the files like the names or something. Very weird. Good thing you had metrics to catch it. I upload all object storage stuff to bunny for live but also to backblace for backup. I’ve always wanted to implement fail over client side for any asset over to bacblaze but seems like a lot of overhead

sergiotapia 7mo ago

I'm a very grateful bunnycdn customer. they are great, lovely UX and great performance and price. we use them to store our image files and other documents.

dgl 7mo ago

I tried using their Magic Containers product and there were issues that showed a lack of attention to detail as well. It's supposed to scale globally (magically!) but I found multiple cases where particular nodes were problematic and the health checks didn't detect them (in fact to start with the health checks didn't even work properly if you had multiple containers, they did fix that). The support was quite slow too, after finding multiple product issues they'd escalate to developers and then come back a month later and ask to retest, but some of this took multiple round trips. I was only using this on a side project, but definitely wouldn't consider them for anything critical, even if they are quite cheap.

tayk47999 7mo ago
[dead]

benatkin 7mo ago

On LowEndTalk: https://lowendtalk.com/discussion/comment/4767400/#Comment_4767400 https://lowendtalk.com/discussion/comment/4767400/#Comment_4...

Volundr 7mo ago

Reply in thread from Bunny: https://old.reddit.com/r/webdev/comments/1sglytg/bunnycdn_has_been_silently_losing_our_production/of7fogs/ https://old.reddit.com/r/webdev/comments/1sglytg/bunnycdn_ha...

benatkin 7mo ago
[flagged]
- john_strinlai 7mo ago
  the full quote is: "does not matter if this is an isolated edge case. Data loss for 15 months with poor support isn't something that can be waved off as 'edge case'. The fact I had to go to Reddit to get someone's attention for this is insane. By the way, I tried reaching out to various 'senior management' personnel via LinkedIn last year and no-one replied. Escalation requests via the Support thread ignored and declined. Very professional of you guys." and that seems pretty damn reasonable of a reply. the fact they had to go to reddit is insane. a sarcastic "very professional of you guys" is pretty tame, given the situation, and not discrediting at all.
- bakugo 7mo ago
  It's a perfectly understandable response to what is essentially just corporate damage control. I'd be pissed too if I was in their position. They've been ignoring the issue for over a year, yet it somehow took them only 4 hours after the reddit post was made to determine that it's "an isolated edge case", even though there's at least one other user reporting the same problem. The comment was clearly written with a focus on saving face rather than properly apologizing for the terrible support.
  - bobdvb 7mo ago
    Yeah. Someone recommended Bunny to me and I looked at it previously based on that but didn't have a use case then. But as someone who regularly recommends vendors for v.big CDN contracts I'm not impressed with their attempts to downplay things.

RobRivera 7mo ago

Handwaving of professionally agreed upon SLAs as an edgecase and providing poor customer support; thats a paddlin

faangguyindia 7mo ago

what's bunny? a CDN by Hug Hefner?

httpsterio 7mo ago

Not a great way to handle the issue originally, but I think the CEO's response is more than fair. I haven't had to contact Bunny's support so it's a bit disappointing to see this type of runaround from the support, but as an actual issue it seems pretty minor in the end. I've been using Bunny for ~four years after looking at video hosting. In the four years we've saved approximately 50 000€ compared to if we'd gone with Cloudflare video streaming service and it's just been rock solid for us.

mort96 7mo ago
The CEO's response is alright, the VP of engineering's response is a bit more worrying in how it mainly tries to downplay the issue.