5 ms·

> [...] we arranged to have five servers shipped to Amsterdam. However, all five servers were lost in the mail. > These parcels were insured with the shipping

> [...] we arranged to have five servers shipped to Amsterdam. However, all five servers were lost in the mail. > These parcels were insured with the shipping provider, but we have been unable to reach the provider for any information regarding the status of the parcels or any resources for filing an insurance claim. > After several months of attempts, we have ultimately had to write these servers off. I obviously do not know the details here and it might be perfectly understandable given the full picture, but this chain of events sounds weird and could have used some further explanation. Because this makes it sound like several thousand dollars worth of important equipment (and possibly important data) went missing and the issue was completely dropped because emails remained unanswered?

drewdevault 2y ago
There was no lost data, and it wasn't just unanswered emails -- several hours on the phone, juggling between subsidiaries on either side of the pond, with no clear responsible party on their end. The customer service for this shipping provider is totally opaque and automated with AI crap throughout. We did eventually get in touch with some humans but they were not ultimately very helpful. We do have general business insurance and will probably file a claim, but we have two overworked and exhausted staff members, a lot of other priorities, and a budget which is already pretty deeply cut from all of these events, and we just don't have the time and energy to duke it out with an opaque megacorporation right now.
- 2pEXgD0fZ5cF 2y ago
  I see, thanks for elaborating!
- strogonoff 2y ago
  If a shipping provider loses shipment and refuses to respect the insurance agreement, it sounds like they should be sued for damages. The damages are much more than the cost of the shipment, as the loss itself also undermines operations and causes a lot of financial downside.
  - ThePowerOfFuet 2y ago
    Lawyers are not free, and the team already has very few CPU cycles to spare.
    - strogonoff 2y ago
      In some jurisdictions, the losing side will have to cover the cost. In some jurisdictions there are also small claims courts, which do not need legal representation. It’s clear that it caused financial distress whereas as happy paying customer I want SourceHut to continue existing.
      - arp242 2y ago
        The reality is that actually getting your rights is often an enormous hassle, in terms of time, money, and/or stress. If you're BigCorp™® with a Lawyer Department then that's not really a problem: you just send it off to the lawyers and continue with your day. If you're a private individual and/or small business: it very quickly becomes a trade-off. So imagine SourceHut sues. And they win. And losing shipping company has to pay all of sourcehut's fees. There is a very real non-zero chance they will just say "lol nope, fuck you". And then what? Basically nothing you can do about it. Even in domestic cases this is true, international cases even more so. In reality a lot of the world runs on goodwill and voluntary adherence to the rules, with not all that much little stopping bad actors from abusing things.
        strogonoff 2y ago
        I part agree, but that goodwill and adherence also exist thanks to people who are willing to exercise their rights. A shipping company that suddenly can’t operate in a profitable country because they are in violation of court order will lose a lot of money; chances are, they like money.
        arp242 2y ago
        Yes, I agree; see e.g Alan Bates for a famous example. But I don't think anyone could be faulted for not being Alan Bates and just moving on with their life.
- mfsch 2y ago
  Does “no lost data” mean that all drives were wiped or simply that there were backups?
  - drewdevault 2y ago
    Both. We restored from backups and touched up the diff out of band. We then had the drives removed and destroyed.
pentagrama 2y ago
> could have used some further explanation They are not obligated to explain what happened in more detail. Your comment sounds like someone speculating about why a celebrity couple broke up when they release a brief public message announcing their separation.
- remram 2y ago
  You were not obligated to comment on GP's take either, and you also sound like you don't like what a celebrity said about their partner. SourceHut are not obligated to post this blog post, but since they did, we are reading it.
remram 2y ago
And this part you quote comes right after a description of how their primary data center failed so badly that they had to completely evacuate it... It does give the impression that they just use cheap unreliable providers across the board. Not necessarily a bad thing, it might be a calculated risk so they can pass on savings to customers. Still, a strong signal there.
- drewdevault 2y ago
  In retrospect, our datacenter partner in the old datacenter was unreliable. But, we had been there from the start, since I was just hosting a personal server there pre-SourceHut, and during the incident they egregiously violated their SLA with us -- we had reason to expect better from them. We're much more confident in our AMS partner, though.

drewdevault 2y ago

There was no lost data, and it wasn't just unanswered emails -- several hours on the phone, juggling between subsidiaries on either side of the pond, with no clear responsible party on their end. The customer service for this shipping provider is totally opaque and automated with AI crap throughout. We did eventually get in touch with some humans but they were not ultimately very helpful. We do have general business insurance and will probably file a claim, but we have two overworked and exhausted staff members, a lot of other priorities, and a budget which is already pretty deeply cut from all of these events, and we just don't have the time and energy to duke it out with an opaque megacorporation right now.

2pEXgD0fZ5cF 2y ago
I see, thanks for elaborating!
strogonoff 2y ago
If a shipping provider loses shipment and refuses to respect the insurance agreement, it sounds like they should be sued for damages. The damages are much more than the cost of the shipment, as the loss itself also undermines operations and causes a lot of financial downside.
- ThePowerOfFuet 2y ago
  Lawyers are not free, and the team already has very few CPU cycles to spare.
  - strogonoff 2y ago
    In some jurisdictions, the losing side will have to cover the cost. In some jurisdictions there are also small claims courts, which do not need legal representation. It’s clear that it caused financial distress whereas as happy paying customer I want SourceHut to continue existing.
    - arp242 2y ago
      The reality is that actually getting your rights is often an enormous hassle, in terms of time, money, and/or stress. If you're BigCorp™® with a Lawyer Department then that's not really a problem: you just send it off to the lawyers and continue with your day. If you're a private individual and/or small business: it very quickly becomes a trade-off. So imagine SourceHut sues. And they win. And losing shipping company has to pay all of sourcehut's fees. There is a very real non-zero chance they will just say "lol nope, fuck you". And then what? Basically nothing you can do about it. Even in domestic cases this is true, international cases even more so. In reality a lot of the world runs on goodwill and voluntary adherence to the rules, with not all that much little stopping bad actors from abusing things.
      - strogonoff 2y ago
        I part agree, but that goodwill and adherence also exist thanks to people who are willing to exercise their rights. A shipping company that suddenly can’t operate in a profitable country because they are in violation of court order will lose a lot of money; chances are, they like money.
        arp242 2y ago
        Yes, I agree; see e.g Alan Bates for a famous example. But I don't think anyone could be faulted for not being Alan Bates and just moving on with their life.
mfsch 2y ago
Does “no lost data” mean that all drives were wiped or simply that there were backups?
- drewdevault 2y ago
  Both. We restored from backups and touched up the diff out of band. We then had the drives removed and destroyed.

pentagrama 2y ago

> could have used some further explanation They are not obligated to explain what happened in more detail. Your comment sounds like someone speculating about why a celebrity couple broke up when they release a brief public message announcing their separation.

remram 2y ago
You were not obligated to comment on GP's take either, and you also sound like you don't like what a celebrity said about their partner. SourceHut are not obligated to post this blog post, but since they did, we are reading it.

remram 2y ago

And this part you quote comes right after a description of how their primary data center failed so badly that they had to completely evacuate it... It does give the impression that they just use cheap unreliable providers across the board. Not necessarily a bad thing, it might be a calculated risk so they can pass on savings to customers. Still, a strong signal there.

drewdevault 2y ago
In retrospect, our datacenter partner in the old datacenter was unreliable. But, we had been there from the start, since I was just hosting a personal server there pre-SourceHut, and during the incident they egregiously violated their SLA with us -- we had reason to expect better from them. We're much more confident in our AMS partner, though.