6 ms·

Paul: I'm surprised you didn't do a scream test. Not everyone is going to see those emails and even those that do may not understand what they are reading. Int

by js2 3y ago

Paul: I'm surprised you didn't do a scream test. Not everyone is going to see those emails and even those that do may not understand what they are reading. Internally at my company we always do scream tests as part of our EOL process because we know we can't reach everyone, even our own employees. https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-scream-test-to-silence-its-unused-servers/ https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-... Fun story: my mortgage got sold last year. Not the first time. I got emails from the old mortgage company and the new mortgage company about the sale, but I skimmed them. I got letters via USPS from the old and new mortgage companies, but I mostly ignored those because 95% of what mortgage companies send me via USPS is junk. So I missed the fact that my automatic payments didn't transfer over. The new mortgage company let me get four months in arrears before they finally FedEx'd me something overnight. That got my attention. I was like: you guys should've FedEx'd me this in the first place. For all they knew, I wasn't getting their emails or letters in the first place because nothing had been sent signature required.

jlund-molfese 3y ago
I even (sort of) do this when I'm deprecating something which my team is the only user of, because sometimes it's hard to tell if something's really unused! First shut off the VPC access while leaving all the other infrastructure and data intact, wait a week or two and see if everything breaks, then get rid of everything else
- deleted 3y ago
  [deleted]
- MarkSweep 3y ago
  There is another variant of this: if you can show that the code you are deleting never worked, there is no need to do a scream test. That is, if anyone cared about the code you are deleting, they would have already been screaming.
  - TeMPOraL 3y ago
    I would be careful with that. Maybe they did scream, but you haven't heard it, and they worked around the issue. Or maybe they did their workaround without saying anything. Or maybe you're wrong about your code not working. It actually may be working in some way that you don't know of, but is useful to someone. To use an ecosystem analogy, once you expose your software to the world beyond your own dev environment, even internally, you'll eventually find that something colonized it - much like everything on this planet that isn't being actively and regularly scrubbed. In my own career, I've seen cases of this. For example, once we were tweaking a little embedded database that supported a half-finished feature meant for internal use, and only then we (as in everyone in the dev team) learned that somehow, the QA & deployment support people got wind of it, and were scripting against exposed parts of that DB for a good year. And, it turns out, it wasn't the only part of the software that we thought of as incidental phenotype (or didn't think of at all), and the other team considered stable behavior. See also the so-called Hyrum's Law: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviours of your system will be depended on by somebody."
    - steveBK123 3y ago
      Re: Hyrum's law This is why user facing code should have (at least) two classes of tests: 1) is it doing what the developer intended it to do 2) is it doing the same thing it did on the last release version with typical user requests? That sounds the same, but it is not. The first class is a set of simpler "happy paths" of intended specific behaviors. The second class is like wargaming. A good way to do this is to replay user requests against your API and see that they return the same results release to release. You may also uncover interesting unintended behavior / conversations to have with users this way.
deleted 3y ago
[deleted]
pmh 3y ago
> scream test I didn't realize there was an actual name for this, so I'll add another for the thread that I haven't seen brought up: email is evil in operations (EIEIO, like the nursery rhyme)
- jacquesm 3y ago
  I think the analogy is to station people at various floors in a big office building while you disconnect (unmarked) cables in a legacy patch panel to see who starts screaming. Then you plug it back in and label the cable...
  - law_enforcement 3y ago
    Or the more general version, where you shutdown a server/service and see if anyone screams. If it's still silent after $days, you can decomission it.
irjustin 3y ago
> I was like: you guys should've FedEx'd me this in the first place. For all they knew, I wasn't getting their emails or letters in the first place because nothing had been sent signature required. I love the scream test, but the analogy you bring up actually - this seems unfair. The cost of Fedex'ing everyone is astounding (for many businesses). But I like the concept. Definitely a sort of "shut off the server for like an hour" and then see who yells. Phone calls for any account that still operating on the location. 100% agreed 3 emails is... hardly anything.
- hoffs 3y ago
  The price doesn't matter in this analogy since it still mostly works
- js2 3y ago
  The point of the story was that if you have something critical to communicate, you can't do it using the same methods that are also used for low-priority mostly junk and expect it to be acted upon. Surely a mortgage servicing company can afford to FedEx an envelope. I wouldn't expect Influx to do so.
  - tracker1 3y ago
    For that matter, registered mail is another option that isn't as expensive as fedex... there's other options that are slightly more costly, and more noticeable than sending via the same structure that junk mail goes through. I expressly don't use automated payments for my mortgage and auto loan(s) as I don't want to have an account miss... I didn't know my mortgage was sold/changed a couple months ago until I went online to change. The same happened with an auto loan a few years back as well.
- jacquesm 3y ago
  When you buy a large, registered debt such as a mortgage the cost of Fedex'ing everyone should be factored into the sale. If that's too much money you shouldn't be buying such assets. Notifying those that are affected properly would seem to be the least you can do in such situations.
pmarreck 3y ago
Something similar happened to me when I lived at a condo in Boston and the management company changed and I somehow missed the memo. I kept sending checks to the old management company for a few months before the problem was discovered. Unfortunately, someone at the old management company apparently had a bit of a gambling habit and petty-thefted that money away, and I never saw it again even though she lost in a lawsuit.
- js2 3y ago
  Semi-related HOA horror story: https://abc11.com/nc-hoa-foreclose-sell-house-woman-didnt-know/12463618/ https://abc11.com/nc-hoa-foreclose-sell-house-woman-didnt-kn...
- Kinrany 3y ago
  Was the employee the defendant, not the company?
  - pmarreck 3y ago
    In this case, yes, it was the employee; I forgot the details as to how/why that was
Axsuul 3y ago
Sounds like the most cost effective way, however.
WirelessGigabit 3y ago
You know, I don't really care that mortgages get sold, but the fact that I need to go out of my way every time to make sure I update my information in their system is astounding.
- jacquesm 3y ago
  Technically that shouldn't even be your problem.
  - TeMPOraL 3y ago
    Neither should be a random third party defrauding a bank, but banks are masters at outsourcing work and responsibility onto their customers, and that's why it's called "identity theft" and suddenly I am at fault somehow.
mrweasel 3y ago
> Not everyone is going to see those emails and even those that do may not understand what they are reading. If that's the case, these companies/people have no business using cloud services. Fair enough that you might not understand the ramification, in that case you contact support. If you don't see those email... that's on you. We operate out of a number of datacenters, they all communicate via email, giving us one month to three notice regarding service windows. If we fail to plan for an outage because we didn't see an email, that's our problem. I don't know why anyone would expect more from a SaaS company. For really large customers, I would assume that they have a customer service representative and yes that person should have called. If you're just a small customer (even if you might be big in your own mind) and just have an account that get billed to a company credit card each month, it's a little naive to think you'd get anything more than an email. We've already seen a number SaaS company just shutting of customers for little to no reason, even AWS has done this. Running things in the cloud is a risk, and it's you job as the operations team to stay on top of things, have backup plan, because you cannot expect cloud vendors to care about some random customer who just signed up using a credit card and a nondescript email. They should, but they don't. A good rule is: Don't expect a SaaS/cloud company to put in more effort contacting you than you did signing up.
- acatton 3y ago
  > > Not everyone is going to see those emails and even those that do may not understand what they are reading. > > If that's the case, these companies/people have no business using cloud services. Cloud services are responsible for this. I've signed up to many cloud services where I purposefully unchecked all the newsletter/updates/... notifications. But I still receive notifications for stuff unrelated to what I use. These emails are full of marketing/PR jargon, where it's unclear whether I'm affected by the change or whether there is even a change! Cloud services are lazy, don't look at their customer use, spam everybody, and blame their customers when they missed an important update due to noise-fatigue. This is the main reason why I stopped using SaaS.
- tracker1 3y ago
  Unless the cloud provider can provide proof that the person received and read such a notice, then they can still be sued for actual damages... and I'd be surprised if that doesn't happen in this case. The fact is, there are many options from a cooldown, scream test, automated backup for migration/recovery... this organization did none of those things and absolutely deserves to lose massively as a result. This is a DATABASE as a Service... RETENTION should be one of the highest priorities. For that matter, it would have been better if they auto-migrated in an OFF status, or otherwise backed up... just hitting the DELETE ALL DATA button is wrong. Several of the posters in the thread indicate they received no such emails.
yard2010 3y ago
Clearly, it's easier to say sorry than do things right up front
- steveBK123 3y ago
  LOL, have they even said sorry though?

by js2 3y ago

jlund-molfese 3y ago

I even (sort of) do this when I'm deprecating something which my team is the only user of, because sometimes it's hard to tell if something's really unused! First shut off the VPC access while leaving all the other infrastructure and data intact, wait a week or two and see if everything breaks, then get rid of everything else

deleted 3y ago
[deleted]
MarkSweep 3y ago
There is another variant of this: if you can show that the code you are deleting never worked, there is no need to do a scream test. That is, if anyone cared about the code you are deleting, they would have already been screaming.
- TeMPOraL 3y ago
  I would be careful with that. Maybe they did scream, but you haven't heard it, and they worked around the issue. Or maybe they did their workaround without saying anything. Or maybe you're wrong about your code not working. It actually may be working in some way that you don't know of, but is useful to someone. To use an ecosystem analogy, once you expose your software to the world beyond your own dev environment, even internally, you'll eventually find that something colonized it - much like everything on this planet that isn't being actively and regularly scrubbed. In my own career, I've seen cases of this. For example, once we were tweaking a little embedded database that supported a half-finished feature meant for internal use, and only then we (as in everyone in the dev team) learned that somehow, the QA & deployment support people got wind of it, and were scripting against exposed parts of that DB for a good year. And, it turns out, it wasn't the only part of the software that we thought of as incidental phenotype (or didn't think of at all), and the other team considered stable behavior. See also the so-called Hyrum's Law: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviours of your system will be depended on by somebody."
  - steveBK123 3y ago
    Re: Hyrum's law This is why user facing code should have (at least) two classes of tests: 1) is it doing what the developer intended it to do 2) is it doing the same thing it did on the last release version with typical user requests? That sounds the same, but it is not. The first class is a set of simpler "happy paths" of intended specific behaviors. The second class is like wargaming. A good way to do this is to replay user requests against your API and see that they return the same results release to release. You may also uncover interesting unintended behavior / conversations to have with users this way.

deleted 3y ago

[deleted]

pmh 3y ago

> scream test I didn't realize there was an actual name for this, so I'll add another for the thread that I haven't seen brought up: email is evil in operations (EIEIO, like the nursery rhyme)

jacquesm 3y ago
I think the analogy is to station people at various floors in a big office building while you disconnect (unmarked) cables in a legacy patch panel to see who starts screaming. Then you plug it back in and label the cable...
- law_enforcement 3y ago
  Or the more general version, where you shutdown a server/service and see if anyone screams. If it's still silent after $days, you can decomission it.

irjustin 3y ago

> I was like: you guys should've FedEx'd me this in the first place. For all they knew, I wasn't getting their emails or letters in the first place because nothing had been sent signature required. I love the scream test, but the analogy you bring up actually - this seems unfair. The cost of Fedex'ing everyone is astounding (for many businesses). But I like the concept. Definitely a sort of "shut off the server for like an hour" and then see who yells. Phone calls for any account that still operating on the location. 100% agreed 3 emails is... hardly anything.

hoffs 3y ago
The price doesn't matter in this analogy since it still mostly works
js2 3y ago
The point of the story was that if you have something critical to communicate, you can't do it using the same methods that are also used for low-priority mostly junk and expect it to be acted upon. Surely a mortgage servicing company can afford to FedEx an envelope. I wouldn't expect Influx to do so.
- tracker1 3y ago
  For that matter, registered mail is another option that isn't as expensive as fedex... there's other options that are slightly more costly, and more noticeable than sending via the same structure that junk mail goes through. I expressly don't use automated payments for my mortgage and auto loan(s) as I don't want to have an account miss... I didn't know my mortgage was sold/changed a couple months ago until I went online to change. The same happened with an auto loan a few years back as well.
jacquesm 3y ago
When you buy a large, registered debt such as a mortgage the cost of Fedex'ing everyone should be factored into the sale. If that's too much money you shouldn't be buying such assets. Notifying those that are affected properly would seem to be the least you can do in such situations.

pmarreck 3y ago

Something similar happened to me when I lived at a condo in Boston and the management company changed and I somehow missed the memo. I kept sending checks to the old management company for a few months before the problem was discovered. Unfortunately, someone at the old management company apparently had a bit of a gambling habit and petty-thefted that money away, and I never saw it again even though she lost in a lawsuit.

js2 3y ago
Semi-related HOA horror story: https://abc11.com/nc-hoa-foreclose-sell-house-woman-didnt-know/12463618/ https://abc11.com/nc-hoa-foreclose-sell-house-woman-didnt-kn...
Kinrany 3y ago
Was the employee the defendant, not the company?
- pmarreck 3y ago
  In this case, yes, it was the employee; I forgot the details as to how/why that was

Axsuul 3y ago

Sounds like the most cost effective way, however.

WirelessGigabit 3y ago

You know, I don't really care that mortgages get sold, but the fact that I need to go out of my way every time to make sure I update my information in their system is astounding.

jacquesm 3y ago
Technically that shouldn't even be your problem.
- TeMPOraL 3y ago
  Neither should be a random third party defrauding a bank, but banks are masters at outsourcing work and responsibility onto their customers, and that's why it's called "identity theft" and suddenly I am at fault somehow.

mrweasel 3y ago

> Not everyone is going to see those emails and even those that do may not understand what they are reading. If that's the case, these companies/people have no business using cloud services. Fair enough that you might not understand the ramification, in that case you contact support. If you don't see those email... that's on you. We operate out of a number of datacenters, they all communicate via email, giving us one month to three notice regarding service windows. If we fail to plan for an outage because we didn't see an email, that's our problem. I don't know why anyone would expect more from a SaaS company. For really large customers, I would assume that they have a customer service representative and yes that person should have called. If you're just a small customer (even if you might be big in your own mind) and just have an account that get billed to a company credit card each month, it's a little naive to think you'd get anything more than an email. We've already seen a number SaaS company just shutting of customers for little to no reason, even AWS has done this. Running things in the cloud is a risk, and it's you job as the operations team to stay on top of things, have backup plan, because you cannot expect cloud vendors to care about some random customer who just signed up using a credit card and a nondescript email. They should, but they don't. A good rule is: Don't expect a SaaS/cloud company to put in more effort contacting you than you did signing up.

acatton 3y ago
> > Not everyone is going to see those emails and even those that do may not understand what they are reading. > > If that's the case, these companies/people have no business using cloud services. Cloud services are responsible for this. I've signed up to many cloud services where I purposefully unchecked all the newsletter/updates/... notifications. But I still receive notifications for stuff unrelated to what I use. These emails are full of marketing/PR jargon, where it's unclear whether I'm affected by the change or whether there is even a change! Cloud services are lazy, don't look at their customer use, spam everybody, and blame their customers when they missed an important update due to noise-fatigue. This is the main reason why I stopped using SaaS.
tracker1 3y ago
Unless the cloud provider can provide proof that the person received and read such a notice, then they can still be sued for actual damages... and I'd be surprised if that doesn't happen in this case. The fact is, there are many options from a cooldown, scream test, automated backup for migration/recovery... this organization did none of those things and absolutely deserves to lose massively as a result. This is a DATABASE as a Service... RETENTION should be one of the highest priorities. For that matter, it would have been better if they auto-migrated in an OFF status, or otherwise backed up... just hitting the DELETE ALL DATA button is wrong. Several of the posters in the thread indicate they received no such emails.

yard2010 3y ago

Clearly, it's easier to say sorry than do things right up front

steveBK123 3y ago
LOL, have they even said sorry though?