8 ms·

Backblaze S3 API is having some downtime

32 points by doctorfoo 6y ago

OJFord 6y ago
Yes! Glad it's not just me. And no status page. (?!)
- zhan_eg 6y ago
  Well, it looks like it is in the time frame of their scheduled maintenance https://www.backblaze.com/scheduled-maintenance.html https://www.backblaze.com/scheduled-maintenance.html -- Disregard, I got AM and PM confused :(
  - jsiepkes 6y ago
    Can't be because this has been going on for over 3 hours now. The timeframe is only 1 hour.
  - OJFord 6y ago
    No, I saw that, but that says 3 pm not am. Pacific time is several hours behind me in the UK and it's 11am here. I don't think it was even Thursday yet Pacific time when this started.
- jsiepkes 6y ago
  I also can't find a status page and not so much as a peep from Backblaze anywhere. I pinged them on Twitter [1] to no avail. I like Backblaze but this has been going on for 3 hours now. You can't be down for 3 hours without any communication about it... [1] https://twitter.com/JasperSiepkes/status/1329347119774375937 https://twitter.com/JasperSiepkes/status/1329347119774375937
  - OJFord 6y ago
    From the support page, I get the impression nothing will happen until 9am Pacific. Tickets have a 1d response time, and chat is Pacific working day with a lunch break. Bottom of https://www.backblaze.com/help.html https://www.backblaze.com/help.html I think they're all sound asleep, blissfully unaware of the day they'll wake up to!
    - slice-beans 6y ago
      It doesn't reflect well on their monitoring/on-call setup that everyone else was paged before them, assuming they are indeed unaware. Perhaps the first they'll know of the problem is when they read it on HN
  - doctorfoo 6y ago
    yup, here desperately migrating to the regular b2 API...
manojlds 6y ago
How prominent is usage of Backblaze and the S3 API in specific? Is HN going to be status page for any random service?
koreanguy 6y ago
https://bgp.he.net/dns/backblazeb2.com#_website https://bgp.he.net/dns/backblazeb2.com#_website This site can’t be reachedCheck if there is a typo in backblazeb.com. If spelling is correct, try running Windows Network Diagnostics. DNS_PROBE_FINISHED_NXDOMAIN
chima3ra 6y ago
well, what can u expect , 4 times cheaper than AWS, 4 times less stability and professionalism. I am now wandering if my data is safe there. I think it will be smart to keep a copy in aws glacier.
- doctorfoo 6y ago
  Feeling kind of the same... I just completed moving my entire app and all user data from AWS. And here we are!
nhoughto 6y ago
Looks like Europe is down too? https://s3.eu-central-003.backblazeb2.com/ https://s3.eu-central-003.backblazeb2.com/
nhoughto 6y ago
But non s3, b2 api is up https://f002.backblazeb2.com/file/blah https://f002.backblazeb2.com/file/blah https://f003.backblazeb2.com/file/blah https://f003.backblazeb2.com/file/blah
mahesh_rm 6y ago
Argh. I believed I was going insane. How can this be the only B2 status page?
atYevP 6y ago
Yev here - we're investing the S3-endpoint issue - apologies for delay in response here, it took them a bit of time to wake me up. *Edit -> we've identified the fix and are rolling it out, you should see S3-Endpoint service resuming shortly!
- mahesh_rm 6y ago
  Happy to know who to bother the next time it happens! :P EDIT: But really, get a simple b2s3api status page up, doesn't need to be anything fancy..
  - atYevP 6y ago
    Well, hopefully there won't be a next time - but also, bothering me doesn't help debug or resolve the problem, our Ops/Tech team do that :D
    - OJFord 6y ago
      I was baffled that I couldn't find a status page at all, but I assumed I was just failing to find it. > Well, hopefully there won't be a next time Are you seriously saying that not only is there not one, but you're against having one because you hope the status is always green?
      - atYevP 6y ago
        No, that comment was edited after I had responded - we're not against status pages - they are on the roadmap, but unfortunately not implemented yet.
        autorun 6y ago
        you just need to update your pixelated loading spinner, everything else is great, thanks for so much, sorry for so little
        OJFord 6y ago
akchou 6y ago
Today's task: starting working on moving back to aws. Storage savings cant compete with the cost of developer time and downtime even when we store a petabyte.

32 points by doctorfoo 6y ago

OJFord 6y ago

Yes! Glad it's not just me. And no status page. (?!)

zhan_eg 6y ago
Well, it looks like it is in the time frame of their scheduled maintenance https://www.backblaze.com/scheduled-maintenance.html https://www.backblaze.com/scheduled-maintenance.html -- Disregard, I got AM and PM confused :(
- jsiepkes 6y ago
  Can't be because this has been going on for over 3 hours now. The timeframe is only 1 hour.
- OJFord 6y ago
  No, I saw that, but that says 3 pm not am. Pacific time is several hours behind me in the UK and it's 11am here. I don't think it was even Thursday yet Pacific time when this started.
jsiepkes 6y ago
I also can't find a status page and not so much as a peep from Backblaze anywhere. I pinged them on Twitter [1] to no avail. I like Backblaze but this has been going on for 3 hours now. You can't be down for 3 hours without any communication about it... [1] https://twitter.com/JasperSiepkes/status/1329347119774375937 https://twitter.com/JasperSiepkes/status/1329347119774375937
- OJFord 6y ago
  From the support page, I get the impression nothing will happen until 9am Pacific. Tickets have a 1d response time, and chat is Pacific working day with a lunch break. Bottom of https://www.backblaze.com/help.html https://www.backblaze.com/help.html I think they're all sound asleep, blissfully unaware of the day they'll wake up to!
  - slice-beans 6y ago
    It doesn't reflect well on their monitoring/on-call setup that everyone else was paged before them, assuming they are indeed unaware. Perhaps the first they'll know of the problem is when they read it on HN
- doctorfoo 6y ago
  yup, here desperately migrating to the regular b2 API...

manojlds 6y ago

How prominent is usage of Backblaze and the S3 API in specific? Is HN going to be status page for any random service?

koreanguy 6y ago

https://bgp.he.net/dns/backblazeb2.com#_website https://bgp.he.net/dns/backblazeb2.com#_website This site can’t be reachedCheck if there is a typo in backblazeb.com. If spelling is correct, try running Windows Network Diagnostics. DNS_PROBE_FINISHED_NXDOMAIN

chima3ra 6y ago

well, what can u expect , 4 times cheaper than AWS, 4 times less stability and professionalism. I am now wandering if my data is safe there. I think it will be smart to keep a copy in aws glacier.

doctorfoo 6y ago
Feeling kind of the same... I just completed moving my entire app and all user data from AWS. And here we are!

nhoughto 6y ago

Looks like Europe is down too? https://s3.eu-central-003.backblazeb2.com/ https://s3.eu-central-003.backblazeb2.com/

nhoughto 6y ago

But non s3, b2 api is up https://f002.backblazeb2.com/file/blah https://f002.backblazeb2.com/file/blah https://f003.backblazeb2.com/file/blah https://f003.backblazeb2.com/file/blah

mahesh_rm 6y ago

Argh. I believed I was going insane. How can this be the only B2 status page?

atYevP 6y ago

Yev here - we're investing the S3-endpoint issue - apologies for delay in response here, it took them a bit of time to wake me up. *Edit -> we've identified the fix and are rolling it out, you should see S3-Endpoint service resuming shortly!

mahesh_rm 6y ago
Happy to know who to bother the next time it happens! :P EDIT: But really, get a simple b2s3api status page up, doesn't need to be anything fancy..
- atYevP 6y ago
  Well, hopefully there won't be a next time - but also, bothering me doesn't help debug or resolve the problem, our Ops/Tech team do that :D
  - OJFord 6y ago
    I was baffled that I couldn't find a status page at all, but I assumed I was just failing to find it. > Well, hopefully there won't be a next time Are you seriously saying that not only is there not one, but you're against having one because you hope the status is always green?
    - atYevP 6y ago
      No, that comment was edited after I had responded - we're not against status pages - they are on the roadmap, but unfortunately not implemented yet.
      - autorun 6y ago
        you just need to update your pixelated loading spinner, everything else is great, thanks for so much, sorry for so little
      - OJFord 6y ago

akchou 6y ago

Today's task: starting working on moving back to aws. Storage savings cant compete with the cost of developer time and downtime even when we store a petabyte.