3 ms·

> Communication gap. First, there was a communication gap between the team that requested the deactivation and the team that ran the deactivation. Instead of pr

by mkl95 5y ago

> Communication gap. First, there was a communication gap between the team that requested the deactivation and the team that ran the deactivation. Instead of providing the IDs of the intended app being marked for deactivation, the team provided the IDs of the entire cloud site where the apps were to be deactivated. So what they are saying is that they are not testing scripts at some staging server before running them in production. It's wild that they've managed to scale their products so much before something like this happened. I hope they've learnt their lesson and they set up some QA process for that stuff.

notdang 5y ago
it seems that it worked as intended, thus they have a QA process. The problem was in the wrong IDs provided and I doubt that at their scale they have a staging environment that duplicates the customer data.
- dylan604 5y ago
  Would it be bad practice to append values to a GUID type of ID that would help a human recognize them? For instance, in this specific case they wanted app IDs as APP-XXXXX-XXXX-blahblah and CLOUD-XXXXX-blahblah. I'm not looking to help their specific problems, but this is more from a general question I've thought of doing but never have done just because I'm sure I'd get laughed at for blazing my own trail
  - bombcar 5y ago
    This is recommended in my experience, but you do have some potential issues when a UUID gets reused or repurposed. WHENEVER a human is involved in the chain, UUIDs can be suspicious because there's no easy way to verify what it is, whereas a human has a good chance of realizing that $1,342.34 is probably not a valid date.
  - xeromal 5y ago
    I kind of dig it. Something that helps make things obvious to a human
  - magicalhippo 5y ago
    While we don't do exactly that, when pulling out lists of ID's like that for someone else, internal or external, we strive to include a description column as well. This might be customer id and customer name, article number and article description, invoice id and invoice number etc. Then it is usually very clear to the recipient what they've been handed. Also, for internal autoinc-type id's, we mostly use sequence generators with non-overlapping "series". That is, we'll start first one at 1 million, second at 2 million or similar. Not perfect but can be useful.
- mkl95 5y ago
  > I doubt that at their scale they have a staging environment that duplicates the customer data. If there is no feasible way of replicating their production environment somewhere else, then there should be some sanity checks in place. Something like "if an abnormally high amount of customer sites go down during the script's execution, kill the script". This is a 20/20 hindsight approach though and if Atlassian engineers can't solve I doubt a random HN user like me can.
- lttlrck 5y ago
  --dry-run and verify the output

by mkl95 5y ago

notdang 5y ago

it seems that it worked as intended, thus they have a QA process. The problem was in the wrong IDs provided and I doubt that at their scale they have a staging environment that duplicates the customer data.

dylan604 5y ago
Would it be bad practice to append values to a GUID type of ID that would help a human recognize them? For instance, in this specific case they wanted app IDs as APP-XXXXX-XXXX-blahblah and CLOUD-XXXXX-blahblah. I'm not looking to help their specific problems, but this is more from a general question I've thought of doing but never have done just because I'm sure I'd get laughed at for blazing my own trail
- bombcar 5y ago
  This is recommended in my experience, but you do have some potential issues when a UUID gets reused or repurposed. WHENEVER a human is involved in the chain, UUIDs can be suspicious because there's no easy way to verify what it is, whereas a human has a good chance of realizing that $1,342.34 is probably not a valid date.
- xeromal 5y ago
  I kind of dig it. Something that helps make things obvious to a human
- magicalhippo 5y ago
  While we don't do exactly that, when pulling out lists of ID's like that for someone else, internal or external, we strive to include a description column as well. This might be customer id and customer name, article number and article description, invoice id and invoice number etc. Then it is usually very clear to the recipient what they've been handed. Also, for internal autoinc-type id's, we mostly use sequence generators with non-overlapping "series". That is, we'll start first one at 1 million, second at 2 million or similar. Not perfect but can be useful.
mkl95 5y ago
> I doubt that at their scale they have a staging environment that duplicates the customer data. If there is no feasible way of replicating their production environment somewhere else, then there should be some sanity checks in place. Something like "if an abnormally high amount of customer sites go down during the script's execution, kill the script". This is a 20/20 hindsight approach though and if Atlassian engineers can't solve I doubt a random HN user like me can.
lttlrck 5y ago
--dry-run and verify the output