11 ms·

How upgrading to Google Apps for Business killed my company's email for 6 hours

143 points by ValentineC 13y ago

dangrossman 13y ago
Rackspace Mail. $2/mo/mailbox. 100% uptime SLA. 24/7/365 phone, email and chat support -- you can call and talk to someone with power to solve your problem at 3AM on Christmas Eve. Big mailboxes, configurable backups, good spam filters. Instant e-mail notifications in your choice of native desktop/mobile clients with IMAP push, or use their hosted webmail. If you aren't fully addicted to the Gmail interface, there are better options for business mailbox hosting.
- Bjoern 13y ago
  100% uptime is technically impossible. For eMail its not even necessary as failure to deliver will result in automatic retry for some time.
  - gte910h 13y ago
    100% uptime SLA doesn't mean it will be up 100%, it means you get paid for every X is isn't up in that period.
- lubos 13y ago
  Or $19/month get a Mailgun.net account (also run by Rackspace) and use it as a redirection service into individual free gmail (or google apps) accounts.
  - dools 13y ago
    I do the same thing for free with my domain registration company and then pop all my messages off gmail using fetchmail for good measure. I then send through an external smtp server. All the fun of gmail, no reliance on Google. I actually also tried to pay them before setting that all up and they couldn't sort it out.
  - thibaut_barrere 13y ago
    Similarly, if you use DNSimple you can enable email-forwarding on your domains [1]. That's the solution I use for my SaaS product so far. [1] http://support.dnsimple.com/articles/email-forwarding http://support.dnsimple.com/articles/email-forwarding
- andybak 13y ago
  No reason not to use a 3rd party IMAP/POP and still use the Gmail front-end. Best of both worlds.
  - mike-cardwell 13y ago
    You're just adding more points of failure.
    - pyre 13y ago
      I thought that a single point of failure was a bad thing. Sure you might want to only have a single place where things can fail, but if that failure is catastrophic...
      - Drakim 13y ago
        No, adding more points of failure doesn't make the first potential point of failure more fail-proof, or anything like that.
      - carbocation 13y ago
        If adding a new layer (gmail) creates a new point of failure without creating any new resilience against failure, you're putting yourself into a bad position. Think about failure like resistance. Points of failure in parallel gives less overall risk of failure. Points of failure in serial (like adding a front-end) gives greater overall risk of failure.
      - nknighthb 13y ago
        "Single point of failure" means a point at which a failure effectively breaks the entire system or workflow. A system may have many SPOFs, which is worse than a single SPOF, which is, in turn, worse than no SPOFs (that is, the desired case is no single failure kills forward progress).
        squidi 13y ago
        Yes, extra points of failure may include: - Having webmail which is not protected with two factor authentication - Enabling POP/IMAP which also does not have two factor authentication
- msy 13y ago
  I've been using Rackspace mail for a few years now, flawless service and shockingly good customer support - helpful, competent people available instantly, cannot recommend enough. Given our email accounts our the skeleton keys of our digital lives relying on a free service, or even a paid service where support is something the company sees as a numbers game seems madness. My only gripe is the lack of support for two-factor authentication, apparently this is under development currently.
  - kngspook 13y ago
    My only gripe with Rackspace is how long it takes for them to push out a new version of Exchange when they're released... Was something like a year and a half after Exchange 2010 was released before it appeared on Rackspace? Curious to see how long before 2013 appears... Otherwise I love them.
- Osmium 13y ago
  I've recently converted to Fastmail.fm, which seems great and they have a nice web interface too. They're now owned by Opera, so I assume their data retention policies must be consistent with EU law, though I think they have servers based in the US so I'm not sure what that means in practice. What I'd like is a service that kept my emails encrypted on-disk with my user password. Not because I think emails are private -- they're not in the sense that I fully expect someone could intercept them in transit, even if that would be illegal -- but just because it would give reassurances that my emails can't be data-mined for advertising purposes à la gmail, or if the mailserver had a major security vulnerability, a hacker couldn't copy all my emails wholesale. But I don't know of any service that provides this, nor of a way to do it myself easily (any mailserver I set up myself would probably have far more vulnerabilities than one I rent from someone else!).
  - JohnTHaller 13y ago
    Unfortunately, most folks would not pay the extra fees that would be necessary for implementing a service like this. And the NSA wouldn't like it as they couldn't just ping Google's server and read all your mail from Gmail.
    - Osmium 13y ago
      I'm paying for Fastmail anyway, so I don't think it's that unreasonable for a paid service.
- voidlogic 13y ago
  Or a $10/mo Digital Ocean VM running Zimbra 8 Community Edition.. https://s3.amazonaws.com/uploads.blog.zimbra.com/wp-content/uploads/2012/09/conversation1.jpeg https://s3.amazonaws.com/uploads.blog.zimbra.com/wp-content/...
- ishbits 13y ago
  Hows the Rackspace web mail interface?
- ozataman 13y ago
  That's great - I love worthy and proper competition combatting nonchalant/tyrant attitude of incumbent giants. It is so annoying to deal with customer service reps that act like they are your boss and you're inconveniencing them through your questions. The stronger the company, the more of those reps you'll get. Thanks for posting this!
- eli 13y ago
  SLA is sort of a red herring here. If my email goes down, getting a partial or full refund on my $2/month mailbox fee is the least of my concerns.
  - e1ven 13y ago
    The point of the SLA here isn't so much to save you the $2, but to make it cost them $2 * N accounts. This gives them an economic incentive to avoid problems.
    - eli 13y ago
      Yeah, fair enough, but regardless of any SLA, I wouldn't expect to be charged full price for inadequate service. Not if you want my continued business anyway.
- rgbrenner 13y ago
  I used to love rackspace... but had a really bad experience recently. I was using Rackspace Files as a CDN for my ecommerce site. One of my customers emailed me saying that the graphics were missing from the site. Turned out 2 of the 3 zones were down, according to status.rackspace.com. They fixed it about 12 hours after it started IIRC. They never sent an email notifying me of the down time. Then -- and this is where they destroyed my trust in their service -- I took a look at the status panel the following day, and they cleared the history of the downtime (it shows 30 days of issues). Instead of showing the issue, they marked it as green and deleted the messages describing the issue. That day, I stopped using Rackspace for everything. (This was about a month ago)
  - rgbrenner 13y ago
    Interesting.. I just looked at their status panel, and the downtime is there, including all of the messages. They were definitely missing the day after it happened. Still disappointed they don't send out emails when there's downtime.. Would probably use them again the future. (This was my one negative experience with them, btw)
bravura 13y ago
tldr; Author experienced an 6-hour outage in Google Apps service, after upgrading to a paid Google Apps account. This behavior is not documented.
- sneak 13y ago
  Is "outage" the right word? That suggests that it's an error, or down for everyone. They really screwed the pooch, here.
  - PommeDeTerre 13y ago
    It's still an "outage" if even just one user is affected. In fact, it's still an "outage" if the service is unavailable (which includes being temporarily unusable) for some period of time, and no users are actively affected by it.
  - 300bps 13y ago
    >Is "outage" the right word? That suggests that it's an error, or down for everyone. As someone who has been in IT for 20 years, I can tell you're in IT. This type of semantics drives business users crazy. Business User: Hey my system is down! IT Person: No, it's not down. Business User: I can't log in to it. IT Person: We are doing scheduled maintenance so the system is unavailable, but it's planned maintenance so it's not down.
    - sneak 13y ago
      Wrong direction. This is not failure-apologetics. Outage suggests temporary and regrettable. This is more like Google just straight up fucking people over via neglect. I was suggesting that stronger negative terms be used.
- darrellsilver 13y ago
  The same thing happened to us as http://www.thinkful.com/ http://www.thinkful.com/, only it took 18 hours to come back! Customer support was hilariously useless as well: "Yes, happens to everyone." "No, there's nothing we can do about it." For us, accounts that were forwarding mail someplace else continued to work, but anyone that had to log into Thinkful's gmail account didn't.
rasterizer 13y ago
So the upgrade took a few hours to complete and it didn't happen instantly in defiance to the laws of physics. Next time play it safe and upgrade during the weekend. The post is dated May 27, is Google planning to announce a new feature for Apps this week and this is some sort of a preemptive PR attack?
- robryan 13y ago
  6 hours... We run a 4 person company and 6 hours during the weekend would still be a problem for us. Given a company larger than 10 might actually upgrade (small amount of shared email accounts) I could see it causing big problems for some.
  - justincormack 13y ago
    Really? Do you run a highly reliable email service yourself then? I certainly would not use gmail if that were the case. You should seriously look at why you are so reliant on email, as it is not really a reliable service, the mail servers at the other side could easily delay mail for 6 hours, it is only a best efforts eventually consistent protocol with very long delays allowed (you don't normally get bounces for days if the mail is queued for retry).
    - thejosh 13y ago
      Well the world of business tends to rely pretty heavily on mail, not sure what other methods of reliable messaging there is? Sure there is phone calls etc, but emails are used when you need an actual record of something. Also relying on mailservers to not bounce the emails and actually deliver in the end is laughable, when mailservers do go down this hardly ever happens properly.
      - justincormack 13y ago
        Well, websites (or APIs) are generally much more reliable forms of messaging, because you can get immediate feedback on processing. The OP said it was business critical; most corporate email is not business critical, and missing business critical email in a channel with spam, high volume unimportant messages, and also relying on a human to answer stuff is all pretty unreliable. Sure you can feed email into a ticketing system, but at least there there is also a web interface (so you do not rely on email). If you need a record of something, I do not think that unsigned emails are legally binding anyway, again you should probably submit signed contracts over the web not email. If it is not time critical, email can work, but the OP said a six hour outage would be a problem, and I still think that is one they have created themselves and is a business risk.
iuguy 13y ago
So the first thing about Google Apps is that you still need to back it up. Because it's (mostly) pretty easy to use it's easy to fall into the trap that it does everything right first time and that your data is safe when you make a change, unlike say running your own infrastructure. It isn't.
surferbayarea 13y ago
We had another issue, where the catch-all address started bouncing. Apparently they have changed the way they handle email routing. So if you upgrade to the 'new google apps', you better be very careful about your email routing rules. Went through a very similar experience with google customer care. Again turned out to be a bug in their system. But what sucks is for 3 days our emails were bouncing! And we're a paid google apps customer...Would be happy to pay 5X, just want some sort of customer service + acceptable SLAs on issues. And yes, please have a test team go over important features like enterprise email! I know it's Google policy to not have features go through a testing team, but maybe they should consider it for Google Apps. /end{rant}
nivla 13y ago
If you are relying on a third-party cloud email service, please keep a regular backup of your data. From random hacking attempts leading to a account lockout to incorrectly being marked for spamming violations, anything could go wrong. The easiest way is to install Thunderbird and let it automatically sync and backup your emails to your computer.
- VLM 13y ago
  From past experience at an email service provider, admittedly a small one, and quite a few years ago, the most likely failure mode related to internet outages knocking out the customer's internal email unlike a locally hosted email server which customers could never seem to understand, and DNS issues (failure to renew, got scammed / domain stolen because of noobs, etc). Probably the single biggest support problem was people (on either side) getting on spammer lists so genuine business communication can no longer flow. Generally speaking the web interface being down, or the imap server being bonkers, was very rarely if ever a problem.
deleted 13y ago
[deleted]
bprater 13y ago
What does it mean to "uninstall Gmail"?
fareesh 13y ago
I've known of a few cases where the Gmail account associated with Google Apps becomes unreachable. Rather than upgrading to the paid plan, most of the people involved just opened a support ticket which was resolved in a few hours.
codev 13y ago
I had a similar situation where I switched a 50 person charity to Google Apps. They had a 30 day trial period and during that time the CEO's credit card was stolen and replaced. I changed the information but the Apps interface said 'No valid payment information stored' so I called their support and they assured me the new card was stored and everything would be fine. Surprise, surprise on the day the account totally shut down. I called in a panic and after a lot of fiddling about (and reassuring me that they had recorded that I thought there might be a problem) they got it back up with everyone's mail after almost a whole day of downtime. To add insult to injury Google don't run a charity program in the UK despite the non-profit laws being a lot stricter than the US and it being easy to validate a charity - so they're paying full rate for Google Apps for Business.
- shrikant 13y ago
  That last part is not entirely accurate. I did some IT volunteer work for a charity very recently, and they definitely had a free Google Apps for Business setup. This is in addition to a reasonably generous grant in the form of AdWords credit.
  - ValentineC 13y ago
    Would you have any idea who we're able to contact about this? I'm involved in a registered UK charity and we were looking to use Google Apps before, but can't justify the monthly cost of $5 per account. Thanks!
    - shrikant 13y ago
      Sorry for the misleading information, chaps. I asked the organisation, and found out that it's not an official programme in the UK. The only one they have here is Google Grants for AdWords - http://www.google.co.uk/grants/ http://www.google.co.uk/grants/. Basically, an enterprising IT volunteer had set them up with vanilla Google Apps back when it was still free for small business and had a 50 user limit, and they just got grandfathered in to the new Apps for Business. They've never had any problems with the user or space limit since they just use the setup for email, analytics and AdWords campaigns and only have about 4-6 people "working" there. Apologies again.
vidyesh 13y ago
tl;dr : Whenever you click on 'Upgrade Now' button just click it and go to sleep. Once you wake you everything should be fine. Author was awake and saw some horrible things.
biot 13y ago
The solution to this is for Google to explicitly tell the end user that the upgrade involves a migration to bigger and better infrastructure, during which their accounts will be inaccessible for a number of hours. Then let the user schedule the migration to minimize the inconvenience. During the migration, show the account status as "Migrating..." instead of empty. Problem solved.
- dpcx 13y ago
  But that would involve Google adding documentation to their system. Which, from my perspective, seems very difficult for them to manage.
- Silhouette 13y ago
  The solution to this is for Google to explicitly tell the end user that the upgrade involves a migration to bigger and better infrastructure, during which their accounts will be inaccessible for a number of hours. [...] Problem solved. Sorry, but that isn't even close to a solution. The problem here is that a business-critical facility was off-line for pretty much an entire business day, at who-knows-what cost to the business. Many small businesses that use facilities like this are trading internationally or otherwise need 24-hour functionality, so the idea that you could just schedule something to "minimize the inconvenience" doesn't really help them at all.
  - cstejerean 13y ago
    Just because it doesn't work for everyone does not mean that a scheduled outage would not be a significant improvement for large percentages of customers. Most businesses can find a 6 hour window.
  - bigiain 13y ago
    I have some clients who are paying MessageLabs something like two orders of magnitude more than Google charges per seat. They would, undertstandably, scream blue murder if their entire corporate email went down for 24hrs. I have other clients who wouldn't dream of paying $5 per user per month, because they get free email forwarding with their domain name or cheapo web hosting. I have _many many_ clients, who if I gave then sufficient notice and said "we're upgrading some infrastructure, we need to take the mail server down for up to 24 hrs, does after close of business the Friday after next suit you, or would you prefer we did it Sat night/Sun morning?", would be _perfectly_ happy. (with the possible exception along the lines of "oh, we're at a trade show that week, can you move it forward or wait till a week later?") I wonder how big the crossover is between companies who really absolutely _have_ to have 24/7/365 email uptime at better tha 3 nines, but who haven't already got something more robust in place than a handful of personal gmail accounts? I can't help but think they're "doin it wrong"…
rexreed 13y ago
I've been POPing my email since the 90s... am I doing it wrong? Yes, accessing email archives on multiple machines is a kinda-headache (I replicate my mailboxes), but never lost email.
- deleted 13y ago
  [deleted]
  - hahainternet 13y ago
    I don't want to contaminate this thread any further but PM me and I'll give you free support.
brokentone 13y ago
Killed it for 6 hours... not convenient, but probably shouldn't make big IT changes during business hours and without warning staff. Our company's move from GApps to Office365 + Outlook has been much more systemically affecting...
- JohnTHaller 13y ago
  I think the issue is that this wasn't a big IT change. It was a simply addition of functionality to a cloud provider, which is SUPPOSED to be easy. Of course, they were using the free version of Gmail, which has no SLA and can be shut off at any time for any reason. Which means they didn't consider email that mission-critical.
- kngspook 13y ago
  How are you liking Office365?
  - brokentone 13y ago
    Hate it.
qwerta 13y ago
And where is your backup? Oh wait you dont have a backup...
xuim21 13y ago
You learned the hard way - don't rely on Google for any sort of commercial service. Their customer support and reliability is abysmal (not to mention notorious and unprecedented levels of privacy abuse).
mrt0mat0 13y ago
I used to be IT manager of a company that wanted to save money and they had me switch them to google business. it was a smooth transition, but the company wanted it's employees (50ish non tech salespeople) to still use Outlook. It was ok. google had an app that allowed this. All was good. I left the company but my friend took my job from me when i left. He now has come across the problem of google not supporting the newest outlook. he gets new computers in and has to downgrade the outlook just to get email to work. headache
- Karunamon 13y ago
  They recently got the Office 2013 connector out there, for what it's worth. Agree that the transition took ENTIRELY too long.
  - mrt0mat0 13y ago
    thanks. i hope he noticed. i'll mention it to him.
joemaller1 13y ago
Different issue with Google Apps: We work with some big pharma companies. During the free evaluation, Google flagged two of our partners and one of our project managers as spammers and silently suspended their accounts. We hosted our email somewhere else.
thracky 13y ago
I migrated a company with 60 franchises from a broken, old, 25 MB per mailbox mail system, to Google Apps last year. The biggest suggestion I have for anyone wanting to use the paid Google Apps is to go through an authorized reseller. You will get better pricing, and they generally have faster access to higher level support than going directly through Google. I had an absolute nightmare of a time with client support (construction workers with little computer literacy) but any legitimate problems during the transition were quickly dealt with by Google via the reseller. Also, good luck getting a response from the Google Apps sales team in the first place.
wc- 13y ago
For whatever it is worth, here is my experience from a few hours ago: my google apps trial ended this morning. Google emailed me, I clicked the billing link in the email and entered my info. After submitting, my account was immediately reactivated and I received a few emails that had arrived while the account was suspended. No downtime or anything... Maybe it has to do with how much data your accounts contain when you switch over to paying for access?
- fpgeek 13y ago
  For a trial, I'd expect you started on the paid infrastructure from day 1, even though you hadn't paid yet. That would mean no migration and no downtime.

143 points by ValentineC 13y ago

Rackspace Mail. $2/mo/mailbox. 100% uptime SLA. 24/7/365 phone, email and chat support -- you can call and talk to someone with power to solve your problem at 3AM on Christmas Eve. Big mailboxes, configurable backups, good spam filters. Instant e-mail notifications in your choice of native desktop/mobile clients with IMAP push, or use their hosted webmail. If you aren't fully addicted to the Gmail interface, there are better options for business mailbox hosting.

Bjoern 13y ago
100% uptime is technically impossible. For eMail its not even necessary as failure to deliver will result in automatic retry for some time.
- gte910h 13y ago
  100% uptime SLA doesn't mean it will be up 100%, it means you get paid for every X is isn't up in that period.
lubos 13y ago
Or $19/month get a Mailgun.net account (also run by Rackspace) and use it as a redirection service into individual free gmail (or google apps) accounts.
- dools 13y ago
  I do the same thing for free with my domain registration company and then pop all my messages off gmail using fetchmail for good measure. I then send through an external smtp server. All the fun of gmail, no reliance on Google. I actually also tried to pay them before setting that all up and they couldn't sort it out.
- thibaut_barrere 13y ago
  Similarly, if you use DNSimple you can enable email-forwarding on your domains [1]. That's the solution I use for my SaaS product so far. [1] http://support.dnsimple.com/articles/email-forwarding http://support.dnsimple.com/articles/email-forwarding
andybak 13y ago
No reason not to use a 3rd party IMAP/POP and still use the Gmail front-end. Best of both worlds.
- mike-cardwell 13y ago
  You're just adding more points of failure.
  - pyre 13y ago
    I thought that a single point of failure was a bad thing. Sure you might want to only have a single place where things can fail, but if that failure is catastrophic...
    - Drakim 13y ago
      No, adding more points of failure doesn't make the first potential point of failure more fail-proof, or anything like that.
    - carbocation 13y ago
      If adding a new layer (gmail) creates a new point of failure without creating any new resilience against failure, you're putting yourself into a bad position. Think about failure like resistance. Points of failure in parallel gives less overall risk of failure. Points of failure in serial (like adding a front-end) gives greater overall risk of failure.
    - nknighthb 13y ago
      "Single point of failure" means a point at which a failure effectively breaks the entire system or workflow. A system may have many SPOFs, which is worse than a single SPOF, which is, in turn, worse than no SPOFs (that is, the desired case is no single failure kills forward progress).
      - squidi 13y ago
        Yes, extra points of failure may include: - Having webmail which is not protected with two factor authentication - Enabling POP/IMAP which also does not have two factor authentication
msy 13y ago
I've been using Rackspace mail for a few years now, flawless service and shockingly good customer support - helpful, competent people available instantly, cannot recommend enough. Given our email accounts our the skeleton keys of our digital lives relying on a free service, or even a paid service where support is something the company sees as a numbers game seems madness. My only gripe is the lack of support for two-factor authentication, apparently this is under development currently.
- kngspook 13y ago
  My only gripe with Rackspace is how long it takes for them to push out a new version of Exchange when they're released... Was something like a year and a half after Exchange 2010 was released before it appeared on Rackspace? Curious to see how long before 2013 appears... Otherwise I love them.
Osmium 13y ago
I've recently converted to Fastmail.fm, which seems great and they have a nice web interface too. They're now owned by Opera, so I assume their data retention policies must be consistent with EU law, though I think they have servers based in the US so I'm not sure what that means in practice. What I'd like is a service that kept my emails encrypted on-disk with my user password. Not because I think emails are private -- they're not in the sense that I fully expect someone could intercept them in transit, even if that would be illegal -- but just because it would give reassurances that my emails can't be data-mined for advertising purposes à la gmail, or if the mailserver had a major security vulnerability, a hacker couldn't copy all my emails wholesale. But I don't know of any service that provides this, nor of a way to do it myself easily (any mailserver I set up myself would probably have far more vulnerabilities than one I rent from someone else!).
- JohnTHaller 13y ago
  Unfortunately, most folks would not pay the extra fees that would be necessary for implementing a service like this. And the NSA wouldn't like it as they couldn't just ping Google's server and read all your mail from Gmail.
  - Osmium 13y ago
    I'm paying for Fastmail anyway, so I don't think it's that unreasonable for a paid service.
voidlogic 13y ago
Or a $10/mo Digital Ocean VM running Zimbra 8 Community Edition.. https://s3.amazonaws.com/uploads.blog.zimbra.com/wp-content/uploads/2012/09/conversation1.jpeg https://s3.amazonaws.com/uploads.blog.zimbra.com/wp-content/...
ishbits 13y ago
Hows the Rackspace web mail interface?
ozataman 13y ago
That's great - I love worthy and proper competition combatting nonchalant/tyrant attitude of incumbent giants. It is so annoying to deal with customer service reps that act like they are your boss and you're inconveniencing them through your questions. The stronger the company, the more of those reps you'll get. Thanks for posting this!
eli 13y ago
SLA is sort of a red herring here. If my email goes down, getting a partial or full refund on my $2/month mailbox fee is the least of my concerns.
- e1ven 13y ago
  The point of the SLA here isn't so much to save you the $2, but to make it cost them $2 * N accounts. This gives them an economic incentive to avoid problems.
  - eli 13y ago
    Yeah, fair enough, but regardless of any SLA, I wouldn't expect to be charged full price for inadequate service. Not if you want my continued business anyway.
rgbrenner 13y ago
I used to love rackspace... but had a really bad experience recently. I was using Rackspace Files as a CDN for my ecommerce site. One of my customers emailed me saying that the graphics were missing from the site. Turned out 2 of the 3 zones were down, according to status.rackspace.com. They fixed it about 12 hours after it started IIRC. They never sent an email notifying me of the down time. Then -- and this is where they destroyed my trust in their service -- I took a look at the status panel the following day, and they cleared the history of the downtime (it shows 30 days of issues). Instead of showing the issue, they marked it as green and deleted the messages describing the issue. That day, I stopped using Rackspace for everything. (This was about a month ago)
- rgbrenner 13y ago
  Interesting.. I just looked at their status panel, and the downtime is there, including all of the messages. They were definitely missing the day after it happened. Still disappointed they don't send out emails when there's downtime.. Would probably use them again the future. (This was my one negative experience with them, btw)

bravura 13y ago

tldr; Author experienced an 6-hour outage in Google Apps service, after upgrading to a paid Google Apps account. This behavior is not documented.

sneak 13y ago
Is "outage" the right word? That suggests that it's an error, or down for everyone. They really screwed the pooch, here.
- PommeDeTerre 13y ago
  It's still an "outage" if even just one user is affected. In fact, it's still an "outage" if the service is unavailable (which includes being temporarily unusable) for some period of time, and no users are actively affected by it.
- 300bps 13y ago
  >Is "outage" the right word? That suggests that it's an error, or down for everyone. As someone who has been in IT for 20 years, I can tell you're in IT. This type of semantics drives business users crazy. Business User: Hey my system is down! IT Person: No, it's not down. Business User: I can't log in to it. IT Person: We are doing scheduled maintenance so the system is unavailable, but it's planned maintenance so it's not down.
  - sneak 13y ago
    Wrong direction. This is not failure-apologetics. Outage suggests temporary and regrettable. This is more like Google just straight up fucking people over via neglect. I was suggesting that stronger negative terms be used.
darrellsilver 13y ago
The same thing happened to us as http://www.thinkful.com/ http://www.thinkful.com/, only it took 18 hours to come back! Customer support was hilariously useless as well: "Yes, happens to everyone." "No, there's nothing we can do about it." For us, accounts that were forwarding mail someplace else continued to work, but anyone that had to log into Thinkful's gmail account didn't.

rasterizer 13y ago

So the upgrade took a few hours to complete and it didn't happen instantly in defiance to the laws of physics. Next time play it safe and upgrade during the weekend. The post is dated May 27, is Google planning to announce a new feature for Apps this week and this is some sort of a preemptive PR attack?

robryan 13y ago
6 hours... We run a 4 person company and 6 hours during the weekend would still be a problem for us. Given a company larger than 10 might actually upgrade (small amount of shared email accounts) I could see it causing big problems for some.
- justincormack 13y ago
  Really? Do you run a highly reliable email service yourself then? I certainly would not use gmail if that were the case. You should seriously look at why you are so reliant on email, as it is not really a reliable service, the mail servers at the other side could easily delay mail for 6 hours, it is only a best efforts eventually consistent protocol with very long delays allowed (you don't normally get bounces for days if the mail is queued for retry).
  - thejosh 13y ago
    Well the world of business tends to rely pretty heavily on mail, not sure what other methods of reliable messaging there is? Sure there is phone calls etc, but emails are used when you need an actual record of something. Also relying on mailservers to not bounce the emails and actually deliver in the end is laughable, when mailservers do go down this hardly ever happens properly.
    - justincormack 13y ago
      Well, websites (or APIs) are generally much more reliable forms of messaging, because you can get immediate feedback on processing. The OP said it was business critical; most corporate email is not business critical, and missing business critical email in a channel with spam, high volume unimportant messages, and also relying on a human to answer stuff is all pretty unreliable. Sure you can feed email into a ticketing system, but at least there there is also a web interface (so you do not rely on email). If you need a record of something, I do not think that unsigned emails are legally binding anyway, again you should probably submit signed contracts over the web not email. If it is not time critical, email can work, but the OP said a six hour outage would be a problem, and I still think that is one they have created themselves and is a business risk.

iuguy 13y ago

So the first thing about Google Apps is that you still need to back it up. Because it's (mostly) pretty easy to use it's easy to fall into the trap that it does everything right first time and that your data is safe when you make a change, unlike say running your own infrastructure. It isn't.

surferbayarea 13y ago

We had another issue, where the catch-all address started bouncing. Apparently they have changed the way they handle email routing. So if you upgrade to the 'new google apps', you better be very careful about your email routing rules. Went through a very similar experience with google customer care. Again turned out to be a bug in their system. But what sucks is for 3 days our emails were bouncing! And we're a paid google apps customer...Would be happy to pay 5X, just want some sort of customer service + acceptable SLAs on issues. And yes, please have a test team go over important features like enterprise email! I know it's Google policy to not have features go through a testing team, but maybe they should consider it for Google Apps. /end{rant}

nivla 13y ago

If you are relying on a third-party cloud email service, please keep a regular backup of your data. From random hacking attempts leading to a account lockout to incorrectly being marked for spamming violations, anything could go wrong. The easiest way is to install Thunderbird and let it automatically sync and backup your emails to your computer.

VLM 13y ago
From past experience at an email service provider, admittedly a small one, and quite a few years ago, the most likely failure mode related to internet outages knocking out the customer's internal email unlike a locally hosted email server which customers could never seem to understand, and DNS issues (failure to renew, got scammed / domain stolen because of noobs, etc). Probably the single biggest support problem was people (on either side) getting on spammer lists so genuine business communication can no longer flow. Generally speaking the web interface being down, or the imap server being bonkers, was very rarely if ever a problem.

deleted 13y ago

[deleted]

bprater 13y ago

What does it mean to "uninstall Gmail"?

fareesh 13y ago

I've known of a few cases where the Gmail account associated with Google Apps becomes unreachable. Rather than upgrading to the paid plan, most of the people involved just opened a support ticket which was resolved in a few hours.

codev 13y ago

I had a similar situation where I switched a 50 person charity to Google Apps. They had a 30 day trial period and during that time the CEO's credit card was stolen and replaced. I changed the information but the Apps interface said 'No valid payment information stored' so I called their support and they assured me the new card was stored and everything would be fine. Surprise, surprise on the day the account totally shut down. I called in a panic and after a lot of fiddling about (and reassuring me that they had recorded that I thought there might be a problem) they got it back up with everyone's mail after almost a whole day of downtime. To add insult to injury Google don't run a charity program in the UK despite the non-profit laws being a lot stricter than the US and it being easy to validate a charity - so they're paying full rate for Google Apps for Business.

shrikant 13y ago
That last part is not entirely accurate. I did some IT volunteer work for a charity very recently, and they definitely had a free Google Apps for Business setup. This is in addition to a reasonably generous grant in the form of AdWords credit.
- ValentineC 13y ago
  Would you have any idea who we're able to contact about this? I'm involved in a registered UK charity and we were looking to use Google Apps before, but can't justify the monthly cost of $5 per account. Thanks!
  - shrikant 13y ago
    Sorry for the misleading information, chaps. I asked the organisation, and found out that it's not an official programme in the UK. The only one they have here is Google Grants for AdWords - http://www.google.co.uk/grants/ http://www.google.co.uk/grants/. Basically, an enterprising IT volunteer had set them up with vanilla Google Apps back when it was still free for small business and had a 50 user limit, and they just got grandfathered in to the new Apps for Business. They've never had any problems with the user or space limit since they just use the setup for email, analytics and AdWords campaigns and only have about 4-6 people "working" there. Apologies again.

vidyesh 13y ago

tl;dr : Whenever you click on 'Upgrade Now' button just click it and go to sleep. Once you wake you everything should be fine. Author was awake and saw some horrible things.

biot 13y ago

The solution to this is for Google to explicitly tell the end user that the upgrade involves a migration to bigger and better infrastructure, during which their accounts will be inaccessible for a number of hours. Then let the user schedule the migration to minimize the inconvenience. During the migration, show the account status as "Migrating..." instead of empty. Problem solved.

dpcx 13y ago
But that would involve Google adding documentation to their system. Which, from my perspective, seems very difficult for them to manage.
Silhouette 13y ago
The solution to this is for Google to explicitly tell the end user that the upgrade involves a migration to bigger and better infrastructure, during which their accounts will be inaccessible for a number of hours. [...] Problem solved. Sorry, but that isn't even close to a solution. The problem here is that a business-critical facility was off-line for pretty much an entire business day, at who-knows-what cost to the business. Many small businesses that use facilities like this are trading internationally or otherwise need 24-hour functionality, so the idea that you could just schedule something to "minimize the inconvenience" doesn't really help them at all.
- cstejerean 13y ago
  Just because it doesn't work for everyone does not mean that a scheduled outage would not be a significant improvement for large percentages of customers. Most businesses can find a 6 hour window.
- bigiain 13y ago
  I have some clients who are paying MessageLabs something like two orders of magnitude more than Google charges per seat. They would, undertstandably, scream blue murder if their entire corporate email went down for 24hrs. I have other clients who wouldn't dream of paying $5 per user per month, because they get free email forwarding with their domain name or cheapo web hosting. I have _many many_ clients, who if I gave then sufficient notice and said "we're upgrading some infrastructure, we need to take the mail server down for up to 24 hrs, does after close of business the Friday after next suit you, or would you prefer we did it Sat night/Sun morning?", would be _perfectly_ happy. (with the possible exception along the lines of "oh, we're at a trade show that week, can you move it forward or wait till a week later?") I wonder how big the crossover is between companies who really absolutely _have_ to have 24/7/365 email uptime at better tha 3 nines, but who haven't already got something more robust in place than a handful of personal gmail accounts? I can't help but think they're "doin it wrong"…

rexreed 13y ago

I've been POPing my email since the 90s... am I doing it wrong? Yes, accessing email archives on multiple machines is a kinda-headache (I replicate my mailboxes), but never lost email.

deleted 13y ago
[deleted]
- hahainternet 13y ago
  I don't want to contaminate this thread any further but PM me and I'll give you free support.

brokentone 13y ago

Killed it for 6 hours... not convenient, but probably shouldn't make big IT changes during business hours and without warning staff. Our company's move from GApps to Office365 + Outlook has been much more systemically affecting...

JohnTHaller 13y ago
I think the issue is that this wasn't a big IT change. It was a simply addition of functionality to a cloud provider, which is SUPPOSED to be easy. Of course, they were using the free version of Gmail, which has no SLA and can be shut off at any time for any reason. Which means they didn't consider email that mission-critical.
kngspook 13y ago
How are you liking Office365?
- brokentone 13y ago
  Hate it.

qwerta 13y ago

And where is your backup? Oh wait you dont have a backup...

xuim21 13y ago

You learned the hard way - don't rely on Google for any sort of commercial service. Their customer support and reliability is abysmal (not to mention notorious and unprecedented levels of privacy abuse).

mrt0mat0 13y ago

I used to be IT manager of a company that wanted to save money and they had me switch them to google business. it was a smooth transition, but the company wanted it's employees (50ish non tech salespeople) to still use Outlook. It was ok. google had an app that allowed this. All was good. I left the company but my friend took my job from me when i left. He now has come across the problem of google not supporting the newest outlook. he gets new computers in and has to downgrade the outlook just to get email to work. headache

Karunamon 13y ago
They recently got the Office 2013 connector out there, for what it's worth. Agree that the transition took ENTIRELY too long.
- mrt0mat0 13y ago
  thanks. i hope he noticed. i'll mention it to him.

joemaller1 13y ago

Different issue with Google Apps: We work with some big pharma companies. During the free evaluation, Google flagged two of our partners and one of our project managers as spammers and silently suspended their accounts. We hosted our email somewhere else.

thracky 13y ago

I migrated a company with 60 franchises from a broken, old, 25 MB per mailbox mail system, to Google Apps last year. The biggest suggestion I have for anyone wanting to use the paid Google Apps is to go through an authorized reseller. You will get better pricing, and they generally have faster access to higher level support than going directly through Google. I had an absolute nightmare of a time with client support (construction workers with little computer literacy) but any legitimate problems during the transition were quickly dealt with by Google via the reseller. Also, good luck getting a response from the Google Apps sales team in the first place.

wc- 13y ago

For whatever it is worth, here is my experience from a few hours ago: my google apps trial ended this morning. Google emailed me, I clicked the billing link in the email and entered my info. After submitting, my account was immediately reactivated and I received a few emails that had arrived while the account was suspended. No downtime or anything... Maybe it has to do with how much data your accounts contain when you switch over to paying for access?

fpgeek 13y ago
For a trial, I'd expect you started on the paid infrastructure from day 1, even though you hadn't paid yet. That would mean no migration and no downtime.