19 ms·

Hi, cofounder and CTO here. We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's

Hi, cofounder and CTO here. We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. If you've been impacted by this, please email me personally and I will do my best to help out: paul at influxdata.com.

flangola7 3y ago
[flagged]
chillfox 3y ago
Email only is not even close to best effort. I know it’s standard to only do email for tech companies, but all other types of companies usually do physical mail and phone calls on top of emails for important notifications. I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.
- ghaff 3y ago
  However, tech companies will often not have your physical address--unlike your bank. And I'd probably block phone calls from some tech company I was a customer of.
  - wongarsu 3y ago
    If they want business customers in Europe they need to create proper invoices, which contain the physical address.
- pauldix 3y ago
  We get an email address because we need to contact our customers. After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss. I realize that shutting down a region isn’t good. It’s not what we would have preferred, but we had to do it for the business. And we made an honest effort to contact all customers to help move them.
  - manquer 3y ago
    As a buyer I have come to expect good vendors to design systems so mistakes (my team or yours) don't cost me sleep or you business.[3] i.e. - they do soft-deletes before hard - have robust access control systems and partitioning - so we don't have to give access to everyone in the org to object model with full r/w - don't instantly nuke the account if a payment goes astray or delayed - try to reach out before to a point of contact before pulling the plug, payment systems can be messy for all sorts of reason, ask before assuming the worst. - customer managers who can connect couple of times a year which usually benefits the vendor as upsells happens on good % of those connects. - also small things like training, certification - Deprecation of service is handled slowly(1 Yr would be expected) and in multiple phases with multiple modes of communication. Not all companies can move fast to plan and execute a major change in location like this in 4 months, bare minimum you would have to consider - End customers (your customer's customers) may need to be notified and may need to sign off - Compliance and GDPR DPA changes - both end customers and internal ones - DR, BCP concerns have to be planned for , not all GCP regions are equivalent. - Documentation and certifications like SoC, ISO, PCI, HIPAA etc usually mean ton of paperwork to modify - SRE/Devops may have to move other services along with telemetry on InfluxDB, may need network whitelisting from their customers, things typically break when moving, need to plan dry runs, rollbacks and so on. A better way to handle service closure would be to shut down but not delete on the planned date[1] , and offer data export separately for few weeks/month after[2]. You can definitely do better than shutting down service and deleting data at the same time . [1] I would do this for internal customers let alone external paying ones [2] You could have even charged for this to offset any costs, most customers wouldn't have a problem paying if they really needed it. [3] Not trying to imply InfluxDB is doing these things, or isn't a good vendor, these are some criteria I have come to measure new vendors by.
    - ssd532 3y ago
      > don't instantly nuke the account if a payment goes astray or delayed Hetzner deleted my server just one week after my payment due date. My credit card failed the payment for some reason. I didn’t notice this because I was ill with Covid. They sent me one email (or at least, I received only one email) as a warning. I only realized the server was gone when my services stopped working. I’m not sure if such a short warning time is common practice among hosting companies, or if it’s unique to Hetzner.
- throwawaysleep 3y ago
  Why do customers believe that they don’t need to read their emails?
  - noveltyaccount 3y ago
    Lmao are you serious? What about emails buried in spam? What if contact x left the company and the emails are black holes? There are a million valid reasons for emails to go poof. "But we emailed you" is weak.
    - throwawaysleep 3y ago
      You actually need to read your spam to check and if a company didn’t bother transition an employee out properly (i.e figure out what their email address was attached to), why is that on the supplier? Why do they need to move mountains so that you can avoid any seriousness about your own operations?
      - SkyPuncher 3y ago
        If a vendor can't properly notify me of major changes, I'm going to find a different vendor. I have far bigger fish to fry than monitoring my inbox for shitty practices.
      - Scarblac 3y ago
        Because their reputation is the most important asset a cloud provider has. You're asking customers to run their business on your computers, after all. Deleting their data first and then complaining that your customers don't run a serious enough operation is not the way to keep the best reputation.
      - wiseowise 3y ago
        Because I’m the customer. > Why do they need to move mountains so that you can avoid any seriousness about your own operations? You won’t stay in business for long with that attitude.
      - flangola7 3y ago
- FooBarWidget 3y ago
  At my company we used to only ask for email address. No names, no phone numbers, no mailing addresses. Because we understood that technical people don't like spam and don't like to give out their data. So we don't ask for them. We also didn't send any reminders for them to check whether their email address was up to date. No account update reminders. To prevent annoying people with spam. So other than sending emails and hoping that they read it, there was nothing else we could do.
  - zuppy 3y ago
    > So other than sending emails and hoping that they read it, there was nothing else we could do. but there are other ways. you can put a big red popup that can only be dismissed by typing "i agree" when the customer logins, you can put the service into read only mode, even with email you can send daily reminders for the last 30 days with a subject like "your data will be deleted in 21 days", etc there are so many things that could have been done.
    - FooBarWidget 3y ago
      What we sold was software that customers deploy locally. We don't have any of their data. But the software would stop working if their license is no longer valid, resulting in downtime. That already made people angry enough. Now we have changed it so that the software never turns off even if license has expired (though it will continue to nag an email address). Updates also cannot be installed.
js2 3y ago
Paul: I'm surprised you didn't do a scream test. Not everyone is going to see those emails and even those that do may not understand what they are reading. Internally at my company we always do scream tests as part of our EOL process because we know we can't reach everyone, even our own employees. https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-scream-test-to-silence-its-unused-servers/ https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-... Fun story: my mortgage got sold last year. Not the first time. I got emails from the old mortgage company and the new mortgage company about the sale, but I skimmed them. I got letters via USPS from the old and new mortgage companies, but I mostly ignored those because 95% of what mortgage companies send me via USPS is junk. So I missed the fact that my automatic payments didn't transfer over. The new mortgage company let me get four months in arrears before they finally FedEx'd me something overnight. That got my attention. I was like: you guys should've FedEx'd me this in the first place. For all they knew, I wasn't getting their emails or letters in the first place because nothing had been sent signature required.
- jlund-molfese 3y ago
  I even (sort of) do this when I'm deprecating something which my team is the only user of, because sometimes it's hard to tell if something's really unused! First shut off the VPC access while leaving all the other infrastructure and data intact, wait a week or two and see if everything breaks, then get rid of everything else
  - deleted 3y ago
    [deleted]
  - MarkSweep 3y ago
    There is another variant of this: if you can show that the code you are deleting never worked, there is no need to do a scream test. That is, if anyone cared about the code you are deleting, they would have already been screaming.
    - TeMPOraL 3y ago
      I would be careful with that. Maybe they did scream, but you haven't heard it, and they worked around the issue. Or maybe they did their workaround without saying anything. Or maybe you're wrong about your code not working. It actually may be working in some way that you don't know of, but is useful to someone. To use an ecosystem analogy, once you expose your software to the world beyond your own dev environment, even internally, you'll eventually find that something colonized it - much like everything on this planet that isn't being actively and regularly scrubbed. In my own career, I've seen cases of this. For example, once we were tweaking a little embedded database that supported a half-finished feature meant for internal use, and only then we (as in everyone in the dev team) learned that somehow, the QA & deployment support people got wind of it, and were scripting against exposed parts of that DB for a good year. And, it turns out, it wasn't the only part of the software that we thought of as incidental phenotype (or didn't think of at all), and the other team considered stable behavior. See also the so-called Hyrum's Law: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviours of your system will be depended on by somebody."
SkyPuncher 3y ago
Wow, that's pretty pathetic and your attitude "we can't help our customers" is even more damning. Email is not reliable enough to simply rely on a few email blasts for this. I would expect: * Those 3 "email blast" notifications. I'm guessing one of two things happened here: * You sent them as an "email blast" from a marketing-type email service. These hit email filters because they came from a known spam IP. * You sent them as a transactional email, but blasted them too quickly and got pegged for spam. Never hit the inbox. * Increasingly common "you haven't migrated emails" if you still detect traffic on these instances. This is pretty critical since some companies might not realize they have affected They should, but things get complex. * Ideally, an automated transfer to another region with automated forwarding. It's okay to have poor performance, but it's not okay to go "poof" entirely. * A soft-delete at the deadline, with 90 to 180 days to finalize migration. If this is costing you dearly, then drive prices up, but don't hard delete data. Frankly, the last one is the real issue. It's literally unbelievable that a database provider didn't soft-delete. Further, I would expect that you'd be able to migrate these to another region to get customers back up an running.
- imglorp 3y ago
  Another problem is that service providers frequently poison the email channel with important sounding engagement dreck and we are now conditioned to ignore it. * Important: migrate to this new feature immediately or you risk missing out!! Vs * Important: migrate your data immediately or you risk losing it!!
  - SkyPuncher 3y ago
    You worded that better than I could have. I ignore most of my vendor's emails because they're simply trying to spam me at this point.
    - xyst 3y ago
      I went to a conference in 2018, gave out my work email. Still get pestered by them
    - 3y ago
mlhpdx 3y ago
Contrary to the majority of the thread here, I find this to be an architectural issue. For whatever reason the system was designed without a way to communicate important service and maintenance issues to the customer. That’s part of the good architectural design of a system – it must include human factors, communication among them.
- ratg13 3y ago
  I’ve worked at companies that aren’t even in the tech sector with less than 10 people and brownouts were SOP. This is just regular old incompetence/negligence/greed.
SentinelRosko 3y ago
This is insane. > We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. What other communication methods were attempted beyond just emails? Big, red obnoxious banners and warnings in various UIs? Phone calls? The fact that it seems as though quite a few customers didn't get your emails, what was the thought process when looking at the workloads that were clearly still active before nuking it from orbit? Or was there no check and it was just assumed that people got the email and migrated? Of the customers who were in that region, how many actually migrated? Was someone tracking these statistics and regularly reporting them to leadership to adjust tactics if there weren't enough migrations or shutdowns happening? This screams either gross incompetence or straight up negligence. This is such a solvable problem (as many here have already mentioned various solutions), but I'm honestly just flabbergasted that this is a problem that is even being discussed here right now. As a DBaaS, the data of your customers should be your number one priority. If its not, y'all need to take a hard look at what the heck your value proposition is. We weren't impact by this directly, but you can be sure that this is going to be one of the topics for discussion amongst my teams this week. Mostly how we can either move off InfluxDB Cloud or ensure that our DR plans are up to date for the rug being pulled out from under us from you guys in the future.
- schoolornot 3y ago
  It seems a banner was added to the UI: https://community.influxdata.com/t/getting-weird-results-from-gcp-europe-west1/30615/12 https://community.influxdata.com/t/getting-weird-results-fro...
  - troupo 3y ago
    It says "The UI was updated with a closure message for these regions." Depends on where and how this message was added. It also means that they had no monitoring in place to see how many people migrated. Edit: They also say that this is reflected on the status page. Here's how their page looks: https://i.imgur.com/xlO4Ik2.png https://i.imgur.com/xlO4Ik2.png Yup. It's literally a green status page that no one would give a second glance. That unreadable white on green? Oh. It's a deprecation message. It even has a subscribe link so that people would immediately and completely dismiss it as an ad due to ad/banner blindness. Edit 2: Someone replied in the thread and added more context for the absolute lack of communication.
white_dragon88 3y ago
[dead]
jasfi 3y ago
Then the title is misleading. Not everyone saw the email, many expected more, e.g. a scream test.
throwaway64478 3y ago
How many times have you said influxdb is about managing the data lifecycle. It is astonishing that you have literally completely ignored one of the primary USPs of your product.
DavidKarlas 3y ago
Why did you feel need to send 3 emails, and not just 1? Is it because you find emails not reliable enough?
jacquesm 3y ago
Hi Paul, email is one-way communication and not guaranteed to be delivered. At a minimum you should have monitored who did and did not respond to the email with some kind of action and those that did not should have more effort expended to be able to reach them. Finally, you should have kept the data for a reasonable amount of time (say 90 days) post shut-down so users that did not get the notification could download it. What you've done is super rude and if I were still a customer in an unaffected region it would definitely be reason enough to leave because it's pointless to sit and wait to see how you'll deal with my data when the time comes. Better to preempt that and leave while I still have control.
arp242 3y ago
> we made our best efforts to notify affected users You call three emails (the last of which was almost 2 months ago) "best efforts"? I had to read your message three times because this is so reality-defying preposterous I just couldn't believe I didn't miss anything. How about warnings on the dashboard? How about an intentional error (or limited service interruption) so that people would log in to their dashboard?
- KomoD 3y ago
  > How about warnings on the dashboard? They did have a warning on the dashboard, problem is a lot of people don't check the dashboard because they don't need to, as they just view everything through grafana, etc. They also had a notice on the status page
  - yencabulator 3y ago
    Getting those people's attention would be what the intentional errors are for.
olliej 3y ago
Multiple comments in the linked issue report not receiving an email. Did you use the same email you use for spam/"marketing" for this notification? The correct course of action is to shutdown the service and give people time to fetch data, not to erase the data as the first indication of shutdown. A few emails are not sufficient if the end result is dataloss, a comment in documentation or release notes is not sufficient (the only reference at least one person in the referenced issue found). truly mind blowing behavior.
asgeirn 3y ago
Former Belgium user here. Checked my inbox, no emails from Influx since June 2022. Then again, I was only using the free tier, so I guess I got what I paid for.
ratg13 3y ago
If you are responsible for this the very least you can do is own up to it and apologize. Trying to assert that you were doing what you thought was right only presents the image that your company is run poorly. The correct thing to do is to admit that your best efforts were not aligned with best practices, and look into remediation. Not “well, we tried”
yard2010 3y ago
Judging from this link, if that's your best effort, I'm afraid to know what it's like when you're slacking :)
dangoodmanUT 3y ago
> I realize that it's not ideal that we've shut down this system Not ideal??? You backed up everyone's DB and moved that to another region so they can just restore and change DB endpoints, right? I don't believe that someone along the chain didn't suggest a scream test or similar. If they did, they must have been ignored.
santafen 3y ago
You could have just responded with ¯\_(ツ)_/¯ and saved a lot of typing.
jpambrun 3y ago
I should not have to say this, but "best efforts" is not enough and is very offensive to every user still relying on your services. You had a duty of results, not merely "best efforts", to reach every-single-one of your active users before hitting shift-delete on their data.
mission_failed 3y ago
Lol great attitude. why would anyone pay money to any company you manage now or in the future if you deliberately trash user data and justify it with 'but we emailed you a couple of times first'
visionik 3y ago
Three emails are not "best efforts". 4.5 months notice is not "best effort". My opinion on best effort: I founded, ran, and sold a SaaS company used by some of the most well known companies in the world. Our "best effort" was a minimum of 12 months notice, with a six month grace period afterwards. Emails weekly. Phone calls at least once a month. Reach out to customer leadership if no response. Then scream test as others suggested.
axman6 3y ago
Paul, are you actually for real right now? Did you really just say "We deleted all your data, and its your fault. We did whisper into the wind three times, you should have heard it. No, there is no chance of recovery"? You might have literally deleted people's whole businesses, companies, who employ real people, who have families, now need to figure out how to continue. Not least of which, your own. If the company survives until Christmas I will be shocked; no one can trust your company ever again - your core business is storing other people's data, and you deleted it, for many, completely without warning. I guess people still use Mongo even after finding it doesn't achieve any property of the CAP theorem, maybe some people will keep using a database provider with a track record of intentionally deleting their paying customers' data. There just aren't enough adjectives for astonishment to adequately describe this situation. I hope you offer Jay Clifford some support, he's clearly been put in the awful situation of having to explain the decisions of others and deliver the awful news. If I were him, I would be in need of serious mental health support, this is an absolutely awful thing to have responsibility for without any ability to rectify.
Gartent 3y ago
You did enough to help, just ignore these ungrateful whiners

flangola7 3y ago

[flagged]

chillfox 3y ago

Email only is not even close to best effort. I know it’s standard to only do email for tech companies, but all other types of companies usually do physical mail and phone calls on top of emails for important notifications. I am not a customer, but it’s really annoying me how tech companies repeatedly think sending emails is somehow anything but the absolute minimum, most lazy option.

ghaff 3y ago
However, tech companies will often not have your physical address--unlike your bank. And I'd probably block phone calls from some tech company I was a customer of.
- wongarsu 3y ago
  If they want business customers in Europe they need to create proper invoices, which contain the physical address.
pauldix 3y ago
We get an email address because we need to contact our customers. After that we make best efforts but if people can’t respond to vendors they pay money to, we’re really at a loss. I realize that shutting down a region isn’t good. It’s not what we would have preferred, but we had to do it for the business. And we made an honest effort to contact all customers to help move them.
- manquer 3y ago
  As a buyer I have come to expect good vendors to design systems so mistakes (my team or yours) don't cost me sleep or you business.[3] i.e. - they do soft-deletes before hard - have robust access control systems and partitioning - so we don't have to give access to everyone in the org to object model with full r/w - don't instantly nuke the account if a payment goes astray or delayed - try to reach out before to a point of contact before pulling the plug, payment systems can be messy for all sorts of reason, ask before assuming the worst. - customer managers who can connect couple of times a year which usually benefits the vendor as upsells happens on good % of those connects. - also small things like training, certification - Deprecation of service is handled slowly(1 Yr would be expected) and in multiple phases with multiple modes of communication. Not all companies can move fast to plan and execute a major change in location like this in 4 months, bare minimum you would have to consider - End customers (your customer's customers) may need to be notified and may need to sign off - Compliance and GDPR DPA changes - both end customers and internal ones - DR, BCP concerns have to be planned for , not all GCP regions are equivalent. - Documentation and certifications like SoC, ISO, PCI, HIPAA etc usually mean ton of paperwork to modify - SRE/Devops may have to move other services along with telemetry on InfluxDB, may need network whitelisting from their customers, things typically break when moving, need to plan dry runs, rollbacks and so on. A better way to handle service closure would be to shut down but not delete on the planned date[1] , and offer data export separately for few weeks/month after[2]. You can definitely do better than shutting down service and deleting data at the same time . [1] I would do this for internal customers let alone external paying ones [2] You could have even charged for this to offset any costs, most customers wouldn't have a problem paying if they really needed it. [3] Not trying to imply InfluxDB is doing these things, or isn't a good vendor, these are some criteria I have come to measure new vendors by.
  - ssd532 3y ago
    > don't instantly nuke the account if a payment goes astray or delayed Hetzner deleted my server just one week after my payment due date. My credit card failed the payment for some reason. I didn’t notice this because I was ill with Covid. They sent me one email (or at least, I received only one email) as a warning. I only realized the server was gone when my services stopped working. I’m not sure if such a short warning time is common practice among hosting companies, or if it’s unique to Hetzner.
throwawaysleep 3y ago
Why do customers believe that they don’t need to read their emails?
- noveltyaccount 3y ago
  Lmao are you serious? What about emails buried in spam? What if contact x left the company and the emails are black holes? There are a million valid reasons for emails to go poof. "But we emailed you" is weak.
  - throwawaysleep 3y ago
    You actually need to read your spam to check and if a company didn’t bother transition an employee out properly (i.e figure out what their email address was attached to), why is that on the supplier? Why do they need to move mountains so that you can avoid any seriousness about your own operations?
    - SkyPuncher 3y ago
      If a vendor can't properly notify me of major changes, I'm going to find a different vendor. I have far bigger fish to fry than monitoring my inbox for shitty practices.
    - Scarblac 3y ago
      Because their reputation is the most important asset a cloud provider has. You're asking customers to run their business on your computers, after all. Deleting their data first and then complaining that your customers don't run a serious enough operation is not the way to keep the best reputation.
    - wiseowise 3y ago
      Because I’m the customer. > Why do they need to move mountains so that you can avoid any seriousness about your own operations? You won’t stay in business for long with that attitude.
    - flangola7 3y ago
FooBarWidget 3y ago
At my company we used to only ask for email address. No names, no phone numbers, no mailing addresses. Because we understood that technical people don't like spam and don't like to give out their data. So we don't ask for them. We also didn't send any reminders for them to check whether their email address was up to date. No account update reminders. To prevent annoying people with spam. So other than sending emails and hoping that they read it, there was nothing else we could do.
- zuppy 3y ago
  > So other than sending emails and hoping that they read it, there was nothing else we could do. but there are other ways. you can put a big red popup that can only be dismissed by typing "i agree" when the customer logins, you can put the service into read only mode, even with email you can send daily reminders for the last 30 days with a subject like "your data will be deleted in 21 days", etc there are so many things that could have been done.
  - FooBarWidget 3y ago
    What we sold was software that customers deploy locally. We don't have any of their data. But the software would stop working if their license is no longer valid, resulting in downtime. That already made people angry enough. Now we have changed it so that the software never turns off even if license has expired (though it will continue to nag an email address). Updates also cannot be installed.

js2 3y ago

Paul: I'm surprised you didn't do a scream test. Not everyone is going to see those emails and even those that do may not understand what they are reading. Internally at my company we always do scream tests as part of our EOL process because we know we can't reach everyone, even our own employees. https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-scream-test-to-silence-its-unused-servers/ https://www.microsoft.com/insidetrack/blog/microsoft-uses-a-... Fun story: my mortgage got sold last year. Not the first time. I got emails from the old mortgage company and the new mortgage company about the sale, but I skimmed them. I got letters via USPS from the old and new mortgage companies, but I mostly ignored those because 95% of what mortgage companies send me via USPS is junk. So I missed the fact that my automatic payments didn't transfer over. The new mortgage company let me get four months in arrears before they finally FedEx'd me something overnight. That got my attention. I was like: you guys should've FedEx'd me this in the first place. For all they knew, I wasn't getting their emails or letters in the first place because nothing had been sent signature required.

jlund-molfese 3y ago
I even (sort of) do this when I'm deprecating something which my team is the only user of, because sometimes it's hard to tell if something's really unused! First shut off the VPC access while leaving all the other infrastructure and data intact, wait a week or two and see if everything breaks, then get rid of everything else
- deleted 3y ago
  [deleted]
- MarkSweep 3y ago
  There is another variant of this: if you can show that the code you are deleting never worked, there is no need to do a scream test. That is, if anyone cared about the code you are deleting, they would have already been screaming.
  - TeMPOraL 3y ago
    I would be careful with that. Maybe they did scream, but you haven't heard it, and they worked around the issue. Or maybe they did their workaround without saying anything. Or maybe you're wrong about your code not working. It actually may be working in some way that you don't know of, but is useful to someone. To use an ecosystem analogy, once you expose your software to the world beyond your own dev environment, even internally, you'll eventually find that something colonized it - much like everything on this planet that isn't being actively and regularly scrubbed. In my own career, I've seen cases of this. For example, once we were tweaking a little embedded database that supported a half-finished feature meant for internal use, and only then we (as in everyone in the dev team) learned that somehow, the QA & deployment support people got wind of it, and were scripting against exposed parts of that DB for a good year. And, it turns out, it wasn't the only part of the software that we thought of as incidental phenotype (or didn't think of at all), and the other team considered stable behavior. See also the so-called Hyrum's Law: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviours of your system will be depended on by somebody."

SkyPuncher 3y ago

Wow, that's pretty pathetic and your attitude "we can't help our customers" is even more damning. Email is not reliable enough to simply rely on a few email blasts for this. I would expect: * Those 3 "email blast" notifications. I'm guessing one of two things happened here: * You sent them as an "email blast" from a marketing-type email service. These hit email filters because they came from a known spam IP. * You sent them as a transactional email, but blasted them too quickly and got pegged for spam. Never hit the inbox. * Increasingly common "you haven't migrated emails" if you still detect traffic on these instances. This is pretty critical since some companies might not realize they have affected They should, but things get complex. * Ideally, an automated transfer to another region with automated forwarding. It's okay to have poor performance, but it's not okay to go "poof" entirely. * A soft-delete at the deadline, with 90 to 180 days to finalize migration. If this is costing you dearly, then drive prices up, but don't hard delete data. Frankly, the last one is the real issue. It's literally unbelievable that a database provider didn't soft-delete. Further, I would expect that you'd be able to migrate these to another region to get customers back up an running.

imglorp 3y ago
Another problem is that service providers frequently poison the email channel with important sounding engagement dreck and we are now conditioned to ignore it. * Important: migrate to this new feature immediately or you risk missing out!! Vs * Important: migrate your data immediately or you risk losing it!!
- SkyPuncher 3y ago
  You worded that better than I could have. I ignore most of my vendor's emails because they're simply trying to spam me at this point.
  - xyst 3y ago
    I went to a conference in 2018, gave out my work email. Still get pestered by them
  - 3y ago

mlhpdx 3y ago

Contrary to the majority of the thread here, I find this to be an architectural issue. For whatever reason the system was designed without a way to communicate important service and maintenance issues to the customer. That’s part of the good architectural design of a system – it must include human factors, communication among them.

ratg13 3y ago
I’ve worked at companies that aren’t even in the tech sector with less than 10 people and brownouts were SOP. This is just regular old incompetence/negligence/greed.

SentinelRosko 3y ago

This is insane. > We notified everyone via email on February 23, April 6 and May 15th. We also offered to help migrate all users. I realize that it's not ideal that we've shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions. What other communication methods were attempted beyond just emails? Big, red obnoxious banners and warnings in various UIs? Phone calls? The fact that it seems as though quite a few customers didn't get your emails, what was the thought process when looking at the workloads that were clearly still active before nuking it from orbit? Or was there no check and it was just assumed that people got the email and migrated? Of the customers who were in that region, how many actually migrated? Was someone tracking these statistics and regularly reporting them to leadership to adjust tactics if there weren't enough migrations or shutdowns happening? This screams either gross incompetence or straight up negligence. This is such a solvable problem (as many here have already mentioned various solutions), but I'm honestly just flabbergasted that this is a problem that is even being discussed here right now. As a DBaaS, the data of your customers should be your number one priority. If its not, y'all need to take a hard look at what the heck your value proposition is. We weren't impact by this directly, but you can be sure that this is going to be one of the topics for discussion amongst my teams this week. Mostly how we can either move off InfluxDB Cloud or ensure that our DR plans are up to date for the rug being pulled out from under us from you guys in the future.

schoolornot 3y ago
It seems a banner was added to the UI: https://community.influxdata.com/t/getting-weird-results-from-gcp-europe-west1/30615/12 https://community.influxdata.com/t/getting-weird-results-fro...
- troupo 3y ago
  It says "The UI was updated with a closure message for these regions." Depends on where and how this message was added. It also means that they had no monitoring in place to see how many people migrated. Edit: They also say that this is reflected on the status page. Here's how their page looks: https://i.imgur.com/xlO4Ik2.png https://i.imgur.com/xlO4Ik2.png Yup. It's literally a green status page that no one would give a second glance. That unreadable white on green? Oh. It's a deprecation message. It even has a subscribe link so that people would immediately and completely dismiss it as an ad due to ad/banner blindness. Edit 2: Someone replied in the thread and added more context for the absolute lack of communication.

white_dragon88 3y ago

[dead]

jasfi 3y ago

Then the title is misleading. Not everyone saw the email, many expected more, e.g. a scream test.

throwaway64478 3y ago

How many times have you said influxdb is about managing the data lifecycle. It is astonishing that you have literally completely ignored one of the primary USPs of your product.

DavidKarlas 3y ago

Why did you feel need to send 3 emails, and not just 1? Is it because you find emails not reliable enough?

jacquesm 3y ago

Hi Paul, email is one-way communication and not guaranteed to be delivered. At a minimum you should have monitored who did and did not respond to the email with some kind of action and those that did not should have more effort expended to be able to reach them. Finally, you should have kept the data for a reasonable amount of time (say 90 days) post shut-down so users that did not get the notification could download it. What you've done is super rude and if I were still a customer in an unaffected region it would definitely be reason enough to leave because it's pointless to sit and wait to see how you'll deal with my data when the time comes. Better to preempt that and leave while I still have control.

arp242 3y ago

> we made our best efforts to notify affected users You call three emails (the last of which was almost 2 months ago) "best efforts"? I had to read your message three times because this is so reality-defying preposterous I just couldn't believe I didn't miss anything. How about warnings on the dashboard? How about an intentional error (or limited service interruption) so that people would log in to their dashboard?

KomoD 3y ago
> How about warnings on the dashboard? They did have a warning on the dashboard, problem is a lot of people don't check the dashboard because they don't need to, as they just view everything through grafana, etc. They also had a notice on the status page
- yencabulator 3y ago
  Getting those people's attention would be what the intentional errors are for.

olliej 3y ago

Multiple comments in the linked issue report not receiving an email. Did you use the same email you use for spam/"marketing" for this notification? The correct course of action is to shutdown the service and give people time to fetch data, not to erase the data as the first indication of shutdown. A few emails are not sufficient if the end result is dataloss, a comment in documentation or release notes is not sufficient (the only reference at least one person in the referenced issue found). truly mind blowing behavior.

asgeirn 3y ago

Former Belgium user here. Checked my inbox, no emails from Influx since June 2022. Then again, I was only using the free tier, so I guess I got what I paid for.

ratg13 3y ago

If you are responsible for this the very least you can do is own up to it and apologize. Trying to assert that you were doing what you thought was right only presents the image that your company is run poorly. The correct thing to do is to admit that your best efforts were not aligned with best practices, and look into remediation. Not “well, we tried”

yard2010 3y ago

Judging from this link, if that's your best effort, I'm afraid to know what it's like when you're slacking :)

dangoodmanUT 3y ago

> I realize that it's not ideal that we've shut down this system Not ideal??? You backed up everyone's DB and moved that to another region so they can just restore and change DB endpoints, right? I don't believe that someone along the chain didn't suggest a scream test or similar. If they did, they must have been ignored.

santafen 3y ago

You could have just responded with ¯\_(ツ)_/¯ and saved a lot of typing.

jpambrun 3y ago

I should not have to say this, but "best efforts" is not enough and is very offensive to every user still relying on your services. You had a duty of results, not merely "best efforts", to reach every-single-one of your active users before hitting shift-delete on their data.

mission_failed 3y ago

Lol great attitude. why would anyone pay money to any company you manage now or in the future if you deliberately trash user data and justify it with 'but we emailed you a couple of times first'

visionik 3y ago

Three emails are not "best efforts". 4.5 months notice is not "best effort". My opinion on best effort: I founded, ran, and sold a SaaS company used by some of the most well known companies in the world. Our "best effort" was a minimum of 12 months notice, with a six month grace period afterwards. Emails weekly. Phone calls at least once a month. Reach out to customer leadership if no response. Then scream test as others suggested.

axman6 3y ago

Paul, are you actually for real right now? Did you really just say "We deleted all your data, and its your fault. We did whisper into the wind three times, you should have heard it. No, there is no chance of recovery"? You might have literally deleted people's whole businesses, companies, who employ real people, who have families, now need to figure out how to continue. Not least of which, your own. If the company survives until Christmas I will be shocked; no one can trust your company ever again - your core business is storing other people's data, and you deleted it, for many, completely without warning. I guess people still use Mongo even after finding it doesn't achieve any property of the CAP theorem, maybe some people will keep using a database provider with a track record of intentionally deleting their paying customers' data. There just aren't enough adjectives for astonishment to adequately describe this situation. I hope you offer Jay Clifford some support, he's clearly been put in the awful situation of having to explain the decisions of others and deliver the awful news. If I were him, I would be in need of serious mental health support, this is an absolutely awful thing to have responsibility for without any ability to rectify.

Gartent 3y ago

You did enough to help, just ignore these ungrateful whiners