12 ms·

I work in a role where half of our stuff is in IBM's cloud (for decisions made before my tenure). On a day in Mid-November, we started receiving alerts from ou

I work in a role where half of our stuff is in IBM's cloud (for decisions made before my tenure). On a day in Mid-November, we started receiving alerts from our UAT and PROD environments. Logged in, all our stuff was gone. Opened a ticket and did some digging and found in our audit logs an IBM SRE had deleted all our stuff. They then told us there was no way to recover and we'd have to rebuild from scratch. They had apparently been doing some "cleanup" and somehow our site number got on to a list. All our servers, attached volumes, subnets, load-balancers were deleted. My boss and I spent the next 30 hours applying terraform and rebuilding anything not automated. Would not recommend IBM Cloud. We are moving off in next 6 months.

robk 5y ago
That seems negligent enough for your company to sue them
- Bombthecat 5y ago
  Good luck suing ibm..
  - Aeolun 5y ago
    It may take a while, but you’ll win.
    - wussboy 5y ago
      At what cost?
      - jcranmer 5y ago
        My guesstimate of the legal fees to try the case would be in the region of $100k.
      - axiosgunnar 5y ago
        In civilised lands the loser has to reimburse the winner's legal costs, so at no cost. If you don't live in civilised lands, well you have other problems.
        jjeaff 5y ago
        I'm not sure that is as civilized as you think it is. Loser pays discourages any small company or individual from suing because the cost is too great if they lose.
        axiosgunnar 5y ago
        But the thread here was talking about a slum dunk case of contract breach.
        kortilla 5y ago
        Even AWS contracts are vague enough to not make this a slam dunk. They pretty much explicitly say if you don’t store stuff in multiple regions you are going to take outages/data loss.
        wussboy 5y ago
        Does your civilized legal system also force the inevitable loser to front your legal fees while the case is ongoing? Or is it possible the plaintiff with deeper pockets can just stretch things out until the legal system has bled you dry and you must withdraw?
    - throwaway98797 5y ago
      pyrrhic victory is not good for business
  - tomrod 5y ago
    For material breach of contract and negligence? Easy peasy.
  - shakna 5y ago
    For breach of contract and negligence, IBM were successfully sued, and then banned from all future projects, by the Queensland government in 2013. [0] Which sets you up with a nice precedent and set of documents to see their angle of attack. > IBM will not be allowed to enter any new contracts with the State Government until it improves its governance and contracting practices. With that ban _still live today_, it astonishes me that any corporation would trust the organisation to actually carry through with their obligations. You have to really, really, royally screw up for a government body to consider you anathema. The old adage of "No one was ever fired for hiring IBM" is no longer true or reasonable. [0] https://statements.qld.gov.au/statements/72961 https://statements.qld.gov.au/statements/72961
    - acdha 5y ago
      > For breach of contract and negligence, IBM were successfully sued, and then banned from all future projects, by the Queensland government in 2013. [0] Which sets you up with a nice precedent and set of documents to see their angle of attack. I mean, yes, but also think about how long a government can afford to have their lawyers pursue a case like that. If you don't have those kind of resources, it's a lot riskier.
      - vidarh 5y ago
        They also have a very significant interest in pursuing it, in that they often have to follow procurement rules that prevent them from excluding vendors without good reason. If you think a vendor will be a problem in the future, getting a legal judgement in place may sometimes be necessary to save a lot of grief that a private company can avoid by just privately and quietly blacklisting the vendor in question.
  - rkk3 5y ago
    Reminds me of the IBM lawyer scene in Halt & Catch Fire
- jiscariot 5y ago
  We're a pretty small org, and not sure we have the organizational heft or resources to do so. That being said, the $1k discount they gave us for Nov based on the broken SLA (resources down) was kind of a slap in the face. We had our minimal core services up in 16hrs, but recorded about 100-120 internal engineering hours for config, testing, and other fallout +1 month. When we opened the initial ticket, the IBM engineer kept saying "you should ask X why they deleted your stuff". Eventually after attaching the LinkedIn page for the IBM SRE in the audit logs, they realized something was screwed up on their side.
JediPig 5y ago
so I worked at IBM cloud, and can confirm this. They bought a cloud service, that by itself, and if left alone would been great. Soft(....something...) was the company... then IBM came and changed everything. I remember the SVP / President of ibm cloud showing up... talking 3 hours about how her son is so great and that we should 'follow' his example... after hour 3, man goes and interrupts and asks a question that was "please get to the point of talking about your son". Few weeks later, massive layoffs, that triggered the warn act. So now I know how IBM works, see first hand meeting, after meeting being totally worthless. I would say stay far away from anything the touch. Just be glad you could move off their platform and not stuck using their platform with their CICD ... else you would be in living hell.
- aunty_helen 5y ago
  I’ve sat in an all hands like that before. Literally 50k$ an hour being wasted listening to some guy we’ll only ever see once talk about his sons 18th birthday. Weird experience.
  - hacknews20 5y ago
    Same here- if they start talking about their kids, it’s over for someone, possibly lots of someones.
  - projektfu 5y ago
    I think it's a power move. Nobody got up and said, "Fuck you, I have better places to be." So they win. You probably thought it was about the business or something.
    - after_care 5y ago
      I have 100% opened up a laptop and tuned out a meeting like that. Never go to a corporate meeting without a laptop.
      - fooqux 5y ago

robk 5y ago

That seems negligent enough for your company to sue them

Bombthecat 5y ago
Good luck suing ibm..
- Aeolun 5y ago
  It may take a while, but you’ll win.
  - wussboy 5y ago
    At what cost?
    - jcranmer 5y ago
      My guesstimate of the legal fees to try the case would be in the region of $100k.
    - axiosgunnar 5y ago
      In civilised lands the loser has to reimburse the winner's legal costs, so at no cost. If you don't live in civilised lands, well you have other problems.
      - jjeaff 5y ago
        I'm not sure that is as civilized as you think it is. Loser pays discourages any small company or individual from suing because the cost is too great if they lose.
        axiosgunnar 5y ago
        But the thread here was talking about a slum dunk case of contract breach.
        kortilla 5y ago
        Even AWS contracts are vague enough to not make this a slam dunk. They pretty much explicitly say if you don’t store stuff in multiple regions you are going to take outages/data loss.
      - wussboy 5y ago
        Does your civilized legal system also force the inevitable loser to front your legal fees while the case is ongoing? Or is it possible the plaintiff with deeper pockets can just stretch things out until the legal system has bled you dry and you must withdraw?
  - throwaway98797 5y ago
    pyrrhic victory is not good for business
- tomrod 5y ago
  For material breach of contract and negligence? Easy peasy.
- shakna 5y ago
  For breach of contract and negligence, IBM were successfully sued, and then banned from all future projects, by the Queensland government in 2013. [0] Which sets you up with a nice precedent and set of documents to see their angle of attack. > IBM will not be allowed to enter any new contracts with the State Government until it improves its governance and contracting practices. With that ban _still live today_, it astonishes me that any corporation would trust the organisation to actually carry through with their obligations. You have to really, really, royally screw up for a government body to consider you anathema. The old adage of "No one was ever fired for hiring IBM" is no longer true or reasonable. [0] https://statements.qld.gov.au/statements/72961 https://statements.qld.gov.au/statements/72961
  - acdha 5y ago
    > For breach of contract and negligence, IBM were successfully sued, and then banned from all future projects, by the Queensland government in 2013. [0] Which sets you up with a nice precedent and set of documents to see their angle of attack. I mean, yes, but also think about how long a government can afford to have their lawyers pursue a case like that. If you don't have those kind of resources, it's a lot riskier.
    - vidarh 5y ago
      They also have a very significant interest in pursuing it, in that they often have to follow procurement rules that prevent them from excluding vendors without good reason. If you think a vendor will be a problem in the future, getting a legal judgement in place may sometimes be necessary to save a lot of grief that a private company can avoid by just privately and quietly blacklisting the vendor in question.
- rkk3 5y ago
  Reminds me of the IBM lawyer scene in Halt & Catch Fire
jiscariot 5y ago
We're a pretty small org, and not sure we have the organizational heft or resources to do so. That being said, the $1k discount they gave us for Nov based on the broken SLA (resources down) was kind of a slap in the face. We had our minimal core services up in 16hrs, but recorded about 100-120 internal engineering hours for config, testing, and other fallout +1 month. When we opened the initial ticket, the IBM engineer kept saying "you should ask X why they deleted your stuff". Eventually after attaching the LinkedIn page for the IBM SRE in the audit logs, they realized something was screwed up on their side.

JediPig 5y ago

so I worked at IBM cloud, and can confirm this. They bought a cloud service, that by itself, and if left alone would been great. Soft(....something...) was the company... then IBM came and changed everything. I remember the SVP / President of ibm cloud showing up... talking 3 hours about how her son is so great and that we should 'follow' his example... after hour 3, man goes and interrupts and asks a question that was "please get to the point of talking about your son". Few weeks later, massive layoffs, that triggered the warn act. So now I know how IBM works, see first hand meeting, after meeting being totally worthless. I would say stay far away from anything the touch. Just be glad you could move off their platform and not stuck using their platform with their CICD ... else you would be in living hell.

aunty_helen 5y ago
I’ve sat in an all hands like that before. Literally 50k$ an hour being wasted listening to some guy we’ll only ever see once talk about his sons 18th birthday. Weird experience.
- hacknews20 5y ago
  Same here- if they start talking about their kids, it’s over for someone, possibly lots of someones.
- projektfu 5y ago
  I think it's a power move. Nobody got up and said, "Fuck you, I have better places to be." So they win. You probably thought it was about the business or something.
  - after_care 5y ago
    I have 100% opened up a laptop and tuned out a meeting like that. Never go to a corporate meeting without a laptop.
    - fooqux 5y ago