3 ms·

For what it's worth, we do have a focus on making sure the hardware works properly etc.; I've always been pushing for that and haven't had any real problems the

For what it's worth, we do have a focus on making sure the hardware works properly etc.; I've always been pushing for that and haven't had any real problems there. This particular board was a test escape. A very unique, thankfully singular (so far as I know) test escape, but there's always a chance for that -- the larger failure in my mind was the breakdown in the support process. With our new controls in place (including rapid escalation / RMA of "odd" faults observed by customer) I am confident this won't happen again. And yes, we're updating tests to catch the newly discovered failure mode. It's an interesting one, the board is a sort of a "zombie" in that if you boot it up by looking at it just right, it'll run stably and pass our stress tests, but it should never have left our factory in that condition due to the other faults. Full stop. :)

tpmx 7y ago
I totally feel your pain. You do so much work to keep things super structured and still bizarre defects escape or only show in the wild. In fact I'm debugging a device in the wild behaving very strangely at this exact moment. > the larger failure in my mind was the breakdown in the support process. 100% agreed. This is where your company failed. And now has redeemed itself.
qubex 7y ago
That particular horror story led me to not equip our development team with a dozen POWER9 machines, which is a pity, because the combination of almost-total trustworthiness and enormous firepower was extremely attractive for our particular niche (plus, I used to hand-code PPC64 AltiVec “velocity engine” code for HFT funds back when Apple sold G5 Xserves). I’m glad to hear that it was a one-off and that you have revised your support strategy, but... really, it made for a pretty frustrating read. It’s not chump change, and going out on that kind of a limb on... trust... which was clearly unwarranted at least at the time of that initial incident. There’s nothing more frustrating than paying top dollar for something exotic and then being stonewalled by oblivious “tech support” when things don’t work as advertised, particularly when something is sufficiently novel or unusual that relatively few other users are out there to proffer help and the potential points of failure are plentiful and “unknown unknowns”. As far as I’m concerned, that incident was extremely bad publicity. That board could’ve been shipped to me. I’d have been left to dangle from my solitary rope and squirm when management asked me what was going on. That’s just not acceptable, sadly.
- deleted 7y ago
  [deleted]
- tpmx 7y ago
  If I were you I'd be a lot more comfortable pulling the trigger after reading all of this. Being worried over what might have been spoilt milk if you had gotten a broken board in the past seems counterproductive.
  - qubex 7y ago
    The new machines I eventually decided to go with were ordered more than a week ago.
    - tpearson-raptor 7y ago
      Sorry to hear that. If you ever get tired of worrying about untrustworthy firmware phoning home, DRM, etc., please consider us for the replacement purchase? ;)
- tpearson-raptor 7y ago
  It also won't happen at this point, ever. Is there anything we can do to help you revisit this decision?
  - qubex 7y ago
    Nope, too late. The decision was made and orders have been placed. You don’t get do-overs, sorry. I might still get a machine for my own amusement, but... I don’t know, reading of some poor guy left dangling for best part of a year just doesn’t build confidence and leaves me with a “there but for the Grace of God go I” feeling, you understand?
    - tpearson-raptor 7y ago
      I understand. I also know this happened to no one else, and it was a pathological case in a support system that has since been completely overhauled. This is one reason why we started the self-serve RMA process -- to show that yes, we are committed to replacing defective hardware in a timely fashion. If the machine is defective, and we can't help via our online troubleshooting guide and Email / phone support, send it right back for replacement. That being said, the assembled systems products are checked with a significant burn-in test etc. before they ship. We have never had an assembled system arrive defective anywhere, except if it was damaged in shipping (i.e. crushed box etc.).

tpmx 7y ago

I totally feel your pain. You do so much work to keep things super structured and still bizarre defects escape or only show in the wild. In fact I'm debugging a device in the wild behaving very strangely at this exact moment. > the larger failure in my mind was the breakdown in the support process. 100% agreed. This is where your company failed. And now has redeemed itself.

qubex 7y ago

That particular horror story led me to not equip our development team with a dozen POWER9 machines, which is a pity, because the combination of almost-total trustworthiness and enormous firepower was extremely attractive for our particular niche (plus, I used to hand-code PPC64 AltiVec “velocity engine” code for HFT funds back when Apple sold G5 Xserves). I’m glad to hear that it was a one-off and that you have revised your support strategy, but... really, it made for a pretty frustrating read. It’s not chump change, and going out on that kind of a limb on... trust... which was clearly unwarranted at least at the time of that initial incident. There’s nothing more frustrating than paying top dollar for something exotic and then being stonewalled by oblivious “tech support” when things don’t work as advertised, particularly when something is sufficiently novel or unusual that relatively few other users are out there to proffer help and the potential points of failure are plentiful and “unknown unknowns”. As far as I’m concerned, that incident was extremely bad publicity. That board could’ve been shipped to me. I’d have been left to dangle from my solitary rope and squirm when management asked me what was going on. That’s just not acceptable, sadly.

deleted 7y ago
[deleted]
tpmx 7y ago
If I were you I'd be a lot more comfortable pulling the trigger after reading all of this. Being worried over what might have been spoilt milk if you had gotten a broken board in the past seems counterproductive.
- qubex 7y ago
  The new machines I eventually decided to go with were ordered more than a week ago.
  - tpearson-raptor 7y ago
    Sorry to hear that. If you ever get tired of worrying about untrustworthy firmware phoning home, DRM, etc., please consider us for the replacement purchase? ;)
tpearson-raptor 7y ago
It also won't happen at this point, ever. Is there anything we can do to help you revisit this decision?
- qubex 7y ago
  Nope, too late. The decision was made and orders have been placed. You don’t get do-overs, sorry. I might still get a machine for my own amusement, but... I don’t know, reading of some poor guy left dangling for best part of a year just doesn’t build confidence and leaves me with a “there but for the Grace of God go I” feeling, you understand?
  - tpearson-raptor 7y ago
    I understand. I also know this happened to no one else, and it was a pathological case in a support system that has since been completely overhauled. This is one reason why we started the self-serve RMA process -- to show that yes, we are committed to replacing defective hardware in a timely fashion. If the machine is defective, and we can't help via our online troubleshooting guide and Email / phone support, send it right back for replacement. That being said, the assembled systems products are checked with a significant burn-in test etc. before they ship. We have never had an assembled system arrive defective anywhere, except if it was damaged in shipping (i.e. crushed box etc.).