3 ms·

If this had happened at some of the other places I have worked, the CEO would have made all the developers stay in the office until the problem was fixed. Why d

by smabie 6y ago

If this had happened at some of the other places I have worked, the CEO would have made all the developers stay in the office until the problem was fixed. Why did no one care?

antipaul 6y ago
Don't have to name them, but what kind of places are these that care? Big, small, tech, non-tech? Locations?
- smabie 6y ago
  Well, for me, it was finance where you know each and everyone of your clients by name because each of them is paying millions of dollars a year for your product. The company had around 50 employees and a 200m valuation.
msh 6y ago
That sounds like pretty bad places to work. And very bad internal processes.
cannam 6y ago
To say no one cared is going too far, but it is interesting to think about why something like this was able to happen. There were various technical and structural factors making it difficult to fix. Weather feeds were known to be problematic (still are I guess) and this code would have been surprisingly low-level C/C++ with custom serialisations and limited logging. Structurally there were problems in getting attention from a team in California to support a problem experienced by a different team in London, especially since it affected relatively few users - a tension in supporting paid products in a company that is focused on non-paying users at far greater scale. (I am assuming this bug would have needed some actual development work - I don't recall, but I think we were familiar enough with common ops problems that it wasn't just a question of kicking one of our own feed servers.) But I do think there was an issue about lack of concern - at heart we didn't have enough confidence in our own product to motivate the personal pain of working through these problems and getting them solved. I think that, if you had gathered us together and asked our collective opinion, we would have suggested that this customer would be better off not using our product at all - it simply wasn't ever likely to be good enough. Once you reach that way of thinking about your own product, it becomes extremely hard to countenance fixing the most difficult problems with it.

by smabie 6y ago

If this had happened at some of the other places I have worked, the CEO would have made all the developers stay in the office until the problem was fixed. Why did no one care?

antipaul 6y ago

Don't have to name them, but what kind of places are these that care? Big, small, tech, non-tech? Locations?

smabie 6y ago
Well, for me, it was finance where you know each and everyone of your clients by name because each of them is paying millions of dollars a year for your product. The company had around 50 employees and a 200m valuation.

msh 6y ago

That sounds like pretty bad places to work. And very bad internal processes.

cannam 6y ago

To say no one cared is going too far, but it is interesting to think about why something like this was able to happen. There were various technical and structural factors making it difficult to fix. Weather feeds were known to be problematic (still are I guess) and this code would have been surprisingly low-level C/C++ with custom serialisations and limited logging. Structurally there were problems in getting attention from a team in California to support a problem experienced by a different team in London, especially since it affected relatively few users - a tension in supporting paid products in a company that is focused on non-paying users at far greater scale. (I am assuming this bug would have needed some actual development work - I don't recall, but I think we were familiar enough with common ops problems that it wasn't just a question of kicking one of our own feed servers.) But I do think there was an issue about lack of concern - at heart we didn't have enough confidence in our own product to motivate the personal pain of working through these problems and getting them solved. I think that, if you had gathered us together and asked our collective opinion, we would have suggested that this customer would be better off not using our product at all - it simply wasn't ever likely to be good enough. Once you reach that way of thinking about your own product, it becomes extremely hard to countenance fixing the most difficult problems with it.