5 ms·

A very old, but still key lesson here for product makers: when you know that you're doing new, early-gen work with a kinda reasonably high probability of someth

by tpmx 7y ago

A very old, but still key lesson here for product makers: when you know that you're doing new, early-gen work with a kinda reasonably high probability of something faulting - you need to provide stellar, high touch support, whatever it costs. And you need to plan to be ready for that from the point when you ship your first thing. (I think this applies both to hardware and software products.) Of course, this is easier said than done, but ultimately this is the core function - are your customers happy or not? As a CTO you need to prepare the company and your investors for this failure mode. You need to be able to explain to them that sometimes making sure that the product actually works and the existing customers are happy is more important than that sexy new feature of the month. It's not always a very easy balance. In my experience, even semi-technical people with a more sales/marketing kind of focus in the company often get blindsided by this. It's on us to prep them for this. It seems like this company learned it the hard way. The key thing is that they did learn.

tpearson-raptor 7y ago
For what it's worth, we do have a focus on making sure the hardware works properly etc.; I've always been pushing for that and haven't had any real problems there. This particular board was a test escape. A very unique, thankfully singular (so far as I know) test escape, but there's always a chance for that -- the larger failure in my mind was the breakdown in the support process. With our new controls in place (including rapid escalation / RMA of "odd" faults observed by customer) I am confident this won't happen again. And yes, we're updating tests to catch the newly discovered failure mode. It's an interesting one, the board is a sort of a "zombie" in that if you boot it up by looking at it just right, it'll run stably and pass our stress tests, but it should never have left our factory in that condition due to the other faults. Full stop. :)
- tpmx 7y ago
  I totally feel your pain. You do so much work to keep things super structured and still bizarre defects escape or only show in the wild. In fact I'm debugging a device in the wild behaving very strangely at this exact moment. > the larger failure in my mind was the breakdown in the support process. 100% agreed. This is where your company failed. And now has redeemed itself.
- qubex 7y ago
  That particular horror story led me to not equip our development team with a dozen POWER9 machines, which is a pity, because the combination of almost-total trustworthiness and enormous firepower was extremely attractive for our particular niche (plus, I used to hand-code PPC64 AltiVec “velocity engine” code for HFT funds back when Apple sold G5 Xserves). I’m glad to hear that it was a one-off and that you have revised your support strategy, but... really, it made for a pretty frustrating read. It’s not chump change, and going out on that kind of a limb on... trust... which was clearly unwarranted at least at the time of that initial incident. There’s nothing more frustrating than paying top dollar for something exotic and then being stonewalled by oblivious “tech support” when things don’t work as advertised, particularly when something is sufficiently novel or unusual that relatively few other users are out there to proffer help and the potential points of failure are plentiful and “unknown unknowns”. As far as I’m concerned, that incident was extremely bad publicity. That board could’ve been shipped to me. I’d have been left to dangle from my solitary rope and squirm when management asked me what was going on. That’s just not acceptable, sadly.
  - deleted 7y ago
    [deleted]
  - tpmx 7y ago
    If I were you I'd be a lot more comfortable pulling the trigger after reading all of this. Being worried over what might have been spoilt milk if you had gotten a broken board in the past seems counterproductive.
    - qubex 7y ago
      The new machines I eventually decided to go with were ordered more than a week ago.
      - tpearson-raptor 7y ago
        Sorry to hear that. If you ever get tired of worrying about untrustworthy firmware phoning home, DRM, etc., please consider us for the replacement purchase? ;)

by tpmx 7y ago

tpearson-raptor 7y ago

For what it's worth, we do have a focus on making sure the hardware works properly etc.; I've always been pushing for that and haven't had any real problems there. This particular board was a test escape. A very unique, thankfully singular (so far as I know) test escape, but there's always a chance for that -- the larger failure in my mind was the breakdown in the support process. With our new controls in place (including rapid escalation / RMA of "odd" faults observed by customer) I am confident this won't happen again. And yes, we're updating tests to catch the newly discovered failure mode. It's an interesting one, the board is a sort of a "zombie" in that if you boot it up by looking at it just right, it'll run stably and pass our stress tests, but it should never have left our factory in that condition due to the other faults. Full stop. :)

tpmx 7y ago
I totally feel your pain. You do so much work to keep things super structured and still bizarre defects escape or only show in the wild. In fact I'm debugging a device in the wild behaving very strangely at this exact moment. > the larger failure in my mind was the breakdown in the support process. 100% agreed. This is where your company failed. And now has redeemed itself.
qubex 7y ago
That particular horror story led me to not equip our development team with a dozen POWER9 machines, which is a pity, because the combination of almost-total trustworthiness and enormous firepower was extremely attractive for our particular niche (plus, I used to hand-code PPC64 AltiVec “velocity engine” code for HFT funds back when Apple sold G5 Xserves). I’m glad to hear that it was a one-off and that you have revised your support strategy, but... really, it made for a pretty frustrating read. It’s not chump change, and going out on that kind of a limb on... trust... which was clearly unwarranted at least at the time of that initial incident. There’s nothing more frustrating than paying top dollar for something exotic and then being stonewalled by oblivious “tech support” when things don’t work as advertised, particularly when something is sufficiently novel or unusual that relatively few other users are out there to proffer help and the potential points of failure are plentiful and “unknown unknowns”. As far as I’m concerned, that incident was extremely bad publicity. That board could’ve been shipped to me. I’d have been left to dangle from my solitary rope and squirm when management asked me what was going on. That’s just not acceptable, sadly.
- deleted 7y ago
  [deleted]
- tpmx 7y ago
  If I were you I'd be a lot more comfortable pulling the trigger after reading all of this. Being worried over what might have been spoilt milk if you had gotten a broken board in the past seems counterproductive.
  - qubex 7y ago
    The new machines I eventually decided to go with were ordered more than a week ago.
    - tpearson-raptor 7y ago
      Sorry to hear that. If you ever get tired of worrying about untrustworthy firmware phoning home, DRM, etc., please consider us for the replacement purchase? ;)