6 ms·

While events like this are not great, isn't it a little myopic to think the solution is to run it yourself? 20 hours of downtime sucks to be sure, but is your o

by darawk 10y ago

While events like this are not great, isn't it a little myopic to think the solution is to run it yourself? 20 hours of downtime sucks to be sure, but is your own uptime rate really going to be better than theirs? Not to mention all the other issues with operating your own service like this.

Scramblejams 10y ago
That comparison isn't apples to apples, though. Internal solutions are frequently less complex than cloud systems, and have fewer and more understandable failure modes as a result. I could throw a rock at my last job and hit mail servers that had better uptime than Gmail, to use one prosaic example.
- krschultz 10y ago
  I literally can't remember the last time Gmail had an outage, and I've been using it for 8 years.
  - Scramblejams 10y ago
    Good! You've been lucky. A few examples from the last 8 years: http://www.pcworld.com/article/160153/google.html http://www.pcworld.com/article/160153/google.html http://news.softpedia.com/news/Longest-Gmail-Meltdown-in-History-Finally-Over-187452.shtml http://news.softpedia.com/news/Longest-Gmail-Meltdown-in-His... http://arstechnica.com/information-technology/2012/12/why-gmail-went-down-google-misconfigured-chromes-sync-server/ http://arstechnica.com/information-technology/2012/12/why-gm... http://www.computerworld.com/article/2486914/web-apps/update--gmail--other-google-services-hit-by-outage-on-friday.html http://www.computerworld.com/article/2486914/web-apps/update... Keep in mind, when comparing it to your experience, is that not all customers are necessarily down when there's a problem. I'm not dumping on Gmail, running a giant system like that has got to be extremely challenging to get right. But let's not wear rose-tinted glasses about the cloud. It can be great, but for some applications you can get better reliability by competent people deploying reliable software on reliable hardware.
    - toomuchtodo 10y ago
      > But let's not wear rose-tinted glasses about the cloud. It can be great, but for some applications you can get better reliability by competent people deploying reliable software on reliable hardware. This doesn't jive with the tech industry hype train though, hence, it being "myopic" to consider running your own services internally.
      - darawk 10y ago
        Of course it's possible to achieve better uptime yourself. The question is not: is it theoretically possible to be more available than gmail. The question is is it cost-effective to do so, once all the relevant factors are considered.
    - apollo5 10y ago
      Wow!! Is there a list of outages on other 'major' services of this magnitude?
  - CapitalistCartr 10y ago
    You may not, but if you run a server your customers will remember like an elephant. http://m.huffpost.com/us/news/gmail-down http://m.huffpost.com/us/news/gmail-down
- tomc1985 10y ago
  I have to agree with this. We recently replaced a homegrown webapp with its MS Dyanmics GP equivalent and headaches have increased exponentially. It's as if the system was designed to be so complex as to require a team of people to operate, plus a manager to supervise, plus a support person to field phone calls... when one guy churned out the original in a day, and ran for a year-and-a-half with only minor technical (and no major organizational) hiccups.
  - jasonjei 10y ago
    When you write of Microsoft Dynamics GP, this FizzBuzz Enterprise Edition repo comes to mind: https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...
  - xyience 10y ago
    > It's as if the system was designed to be so complex as to require a team of people to operate, plus a manager to supervise, plus a support person to field phone calls This is exactly it. Even if not intentionally designed, it ends up that way. As andrewstuart2 mentioned above, "[customers] also got the full expertise of the salesforce engineering organization (which clearly is far from incompetent) being brought to bear on the problem, at no extra charge to them as a customer". This is great I guess, but if your home-grown solution doesn't use, say, Oracle, to start out with, you're going to eliminate a heck of a lot of necessary engineering expertise when things go wrong. Break up systems even more and suddenly just one generalist can do it all on the side...
  - dano 10y ago
    A classic build vs. buy situation where the purchase seems like the solution and in reality, it is a purchase plus development plus support. In many cases the internally designed, function specific, application is better because of its constrained functionality. Many companies, mine included, have made the mistake of purchasing a silver bullet rather than forging one with our own time, talent, and treasure.
- bunderbunder 10y ago
  For many companies, especially smaller ones, the cost of setting up and managing infrastructure that can support high uptimes just isn't manageable. It's not just the hardware, it's also the price of hiring or contracting people with the expertise to do it. Case in point: It's informative that you refer to "mail servers". The plural makes it clear that the company in question isn't in the same league, size-wise, as the target market for most SaaS vendors.
  - Scramblejams 10y ago
    Depends. If you're hoping for high uptimes but aren't forced to guarantee them, you can get awfully far with a good Lenovo box running RAID1, Qmail and judicious application of uninterruptible power supplies.
    - bunderbunder 10y ago
      If you're just after high uptimes for your email server, and your email server doesn't need to be anything more than an email server, and you already happen to have a need for a Unix server and a qualified Unix admin, then sure. Typical companies, though, are mostly Windows shops and really are looking to get something more along the lines of an integrated communication/collaboration/scheduling tool like Microsoft Exchange or GMail.
- acdha 10y ago
  I've also seen internal systems become far more complicated because there was no check on what a project owner could request and cloud systems drive simplicity because e.g. you couldn't try to force Amazon to implement some weird custom behaviour one of your legacy app development teams thought they needed. The general trend I follow is that you're good at things you do regularly. That means that internal systems tend to be much worse at recovering from failure (i.e. the time to recover can be days or even weeks) unless they are careful to design with redundancy in mind and practice it regularly. That's a time-honored tradition in large IT shops but it's also one which often requires constant effort to justify the "wasteful" expense even in places which should know better.
- darawk 10y ago
  Ya that's a fair point. I still think that in general you're likely to have worse uptime, or at best just comparable uptime to large cloud services in most cases. Cloud services just have to be equal to internally run ones in this regard, since they save so much in other areas. So I guess people have to decide: is 20 hours of downtime every few years worth running your own services internally and paying people full time to manage them?
andrewstuart2 10y ago
Agreed. The following statement the author makes is too generalized in my opinion: > What we seem to forget all too easily is that the cloud is, in its simplest form, just someone else’s computer. This isn't even actually true for Infrastructure as a Sevice (IaaS), in which case there's significant intellectual capital that's been invested in the automated provisioning of resources for customers. This is much more true with SaaS products. In this case, the customers of #NA14 did experience downtime. But they also got the full expertise of the salesforce engineering organization (which clearly is far from incompetent) being brought to bear on the problem, at no extra charge to them as a customer, as well as having likely already mitigated instances of this problem before. These prior mitigations, of course, are met with much less fanfare than an outage. It's hard to appreciate things you don't experience.
- stephenr 10y ago
  > > What we seem to forget all too easily is that the cloud is, in its simplest form, just someone else’s computer. > This isn't even actually true for Infrastructure as a Sevice (IaaS), in which case there's significant intellectual capital that's been invested in the automated provisioning of resources for customers. This is much more true with SaaS products. Does all the "intellectual capital" (read: aws/etc runs a highly complex system made up of various parts that sometimes all just shit themselves when a single low level thing hiccups) make it somehow not someone else's computer? The point of "it's just someone else's computer" isn't that it's just a computer like you or I have, it's that it's not some magical, mythical beast. The greatest achievement of aws/et-al is convincing developers they will need "web scale" infrastructure at the drop of a hat.
  - andrewstuart2 10y ago
    What I'm saying is that you can't justifiably say that it's "just someone else's computer" (emphasis mine). Just someone else's computer would be ssh root access to a physical machine in some data center. No tenancy separation, no dynamic allocation, no exposure of failure tolerances (e.g. AWS Availability Zones), etc. To illustrate this point succinctly, there are probably thousands of "somebody else's computers" sitting in a dell warehouse somewhere that are no good to me. The cloud is only useful if there are additional services to help me consume that computer. I'm not saying it doesn't run on somebody else's computer. I could, though, since private clouds are in fact "clouds" that are not someone else's computer. Again, what I'm saying is that it is not a sufficiently specific generalization to accurately describe "cloud." The definition of "cloud" is fundamentally more complex than "just somebody else's computer."
devishard 10y ago
> While events like this are not great, isn't it a little myopic to think the solution is to run it yourself? 20 hours of downtime sucks to be sure, but is your own uptime rate really going to be better than theirs? It sounds very much like you're assuming the answer is no. In my entire 10+ year career I've never had a service I was working on have a 20 hour outage. So, probably, the answer is yes.
- Xylakant 10y ago
  neither did I. but on average, I'm sceptical if my track record is better than theirs, especially on the more complex installations. I've seen cases where a single downed router chopped of a whole building of business people who - since everything was on network drives - couldn't do anything better than play solitaire a whole business day long.
- jjbiotech 10y ago
  > In my entire 10+ year career I've never had a service I was working on have a 20 hour outage. So, probably, the answer is yes. How arrogant. I seriously doubt the services you've managed have even served a fraction of the data and customers SalesForce has.
  - john_fushi 10y ago
    I think his point is that when comparing running SalesForce on the cloud for every users SalesForce has, vs running SalesForce locally for your users, there will a lot less challenge to keep it running locally than on the cloud.
  - bisby 10y ago
    The point isn't to serve the volume that SalesForce has, the point is to serve yourself. If you have experience enough to prevent downtime or to handle it when it happens, then perhaps that is better than risking that to someone else. the 20 hours of downtime was probably BECAUSE of the amount of data/customers to handle. whereas if there is 1 customer, recovering isn't nearly as difficult or time consuming.
  - 10y ago
dangerboysteve 10y ago
Totally agree. The title is click bait to start and the entire premise of the article is self-serving bullshit.
spelunker 10y ago
I was thinking the same thing. Just like the article said, the cloud is someone else's computer. So why do you think your computer woudln't have these issues?

by darawk 10y ago

Scramblejams 10y ago

That comparison isn't apples to apples, though. Internal solutions are frequently less complex than cloud systems, and have fewer and more understandable failure modes as a result. I could throw a rock at my last job and hit mail servers that had better uptime than Gmail, to use one prosaic example.

krschultz 10y ago
I literally can't remember the last time Gmail had an outage, and I've been using it for 8 years.
- Scramblejams 10y ago
  Good! You've been lucky. A few examples from the last 8 years: http://www.pcworld.com/article/160153/google.html http://www.pcworld.com/article/160153/google.html http://news.softpedia.com/news/Longest-Gmail-Meltdown-in-History-Finally-Over-187452.shtml http://news.softpedia.com/news/Longest-Gmail-Meltdown-in-His... http://arstechnica.com/information-technology/2012/12/why-gmail-went-down-google-misconfigured-chromes-sync-server/ http://arstechnica.com/information-technology/2012/12/why-gm... http://www.computerworld.com/article/2486914/web-apps/update--gmail--other-google-services-hit-by-outage-on-friday.html http://www.computerworld.com/article/2486914/web-apps/update... Keep in mind, when comparing it to your experience, is that not all customers are necessarily down when there's a problem. I'm not dumping on Gmail, running a giant system like that has got to be extremely challenging to get right. But let's not wear rose-tinted glasses about the cloud. It can be great, but for some applications you can get better reliability by competent people deploying reliable software on reliable hardware.
  - toomuchtodo 10y ago
    > But let's not wear rose-tinted glasses about the cloud. It can be great, but for some applications you can get better reliability by competent people deploying reliable software on reliable hardware. This doesn't jive with the tech industry hype train though, hence, it being "myopic" to consider running your own services internally.
    - darawk 10y ago
      Of course it's possible to achieve better uptime yourself. The question is not: is it theoretically possible to be more available than gmail. The question is is it cost-effective to do so, once all the relevant factors are considered.
  - apollo5 10y ago
    Wow!! Is there a list of outages on other 'major' services of this magnitude?
- CapitalistCartr 10y ago
  You may not, but if you run a server your customers will remember like an elephant. http://m.huffpost.com/us/news/gmail-down http://m.huffpost.com/us/news/gmail-down
tomc1985 10y ago
I have to agree with this. We recently replaced a homegrown webapp with its MS Dyanmics GP equivalent and headaches have increased exponentially. It's as if the system was designed to be so complex as to require a team of people to operate, plus a manager to supervise, plus a support person to field phone calls... when one guy churned out the original in a day, and ran for a year-and-a-half with only minor technical (and no major organizational) hiccups.
- jasonjei 10y ago
  When you write of Microsoft Dynamics GP, this FizzBuzz Enterprise Edition repo comes to mind: https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...
- xyience 10y ago
  > It's as if the system was designed to be so complex as to require a team of people to operate, plus a manager to supervise, plus a support person to field phone calls This is exactly it. Even if not intentionally designed, it ends up that way. As andrewstuart2 mentioned above, "[customers] also got the full expertise of the salesforce engineering organization (which clearly is far from incompetent) being brought to bear on the problem, at no extra charge to them as a customer". This is great I guess, but if your home-grown solution doesn't use, say, Oracle, to start out with, you're going to eliminate a heck of a lot of necessary engineering expertise when things go wrong. Break up systems even more and suddenly just one generalist can do it all on the side...
- dano 10y ago
  A classic build vs. buy situation where the purchase seems like the solution and in reality, it is a purchase plus development plus support. In many cases the internally designed, function specific, application is better because of its constrained functionality. Many companies, mine included, have made the mistake of purchasing a silver bullet rather than forging one with our own time, talent, and treasure.
bunderbunder 10y ago
For many companies, especially smaller ones, the cost of setting up and managing infrastructure that can support high uptimes just isn't manageable. It's not just the hardware, it's also the price of hiring or contracting people with the expertise to do it. Case in point: It's informative that you refer to "mail servers". The plural makes it clear that the company in question isn't in the same league, size-wise, as the target market for most SaaS vendors.
- Scramblejams 10y ago
  Depends. If you're hoping for high uptimes but aren't forced to guarantee them, you can get awfully far with a good Lenovo box running RAID1, Qmail and judicious application of uninterruptible power supplies.
  - bunderbunder 10y ago
    If you're just after high uptimes for your email server, and your email server doesn't need to be anything more than an email server, and you already happen to have a need for a Unix server and a qualified Unix admin, then sure. Typical companies, though, are mostly Windows shops and really are looking to get something more along the lines of an integrated communication/collaboration/scheduling tool like Microsoft Exchange or GMail.
acdha 10y ago
I've also seen internal systems become far more complicated because there was no check on what a project owner could request and cloud systems drive simplicity because e.g. you couldn't try to force Amazon to implement some weird custom behaviour one of your legacy app development teams thought they needed. The general trend I follow is that you're good at things you do regularly. That means that internal systems tend to be much worse at recovering from failure (i.e. the time to recover can be days or even weeks) unless they are careful to design with redundancy in mind and practice it regularly. That's a time-honored tradition in large IT shops but it's also one which often requires constant effort to justify the "wasteful" expense even in places which should know better.
darawk 10y ago
Ya that's a fair point. I still think that in general you're likely to have worse uptime, or at best just comparable uptime to large cloud services in most cases. Cloud services just have to be equal to internally run ones in this regard, since they save so much in other areas. So I guess people have to decide: is 20 hours of downtime every few years worth running your own services internally and paying people full time to manage them?

andrewstuart2 10y ago

Agreed. The following statement the author makes is too generalized in my opinion: > What we seem to forget all too easily is that the cloud is, in its simplest form, just someone else’s computer. This isn't even actually true for Infrastructure as a Sevice (IaaS), in which case there's significant intellectual capital that's been invested in the automated provisioning of resources for customers. This is much more true with SaaS products. In this case, the customers of #NA14 did experience downtime. But they also got the full expertise of the salesforce engineering organization (which clearly is far from incompetent) being brought to bear on the problem, at no extra charge to them as a customer, as well as having likely already mitigated instances of this problem before. These prior mitigations, of course, are met with much less fanfare than an outage. It's hard to appreciate things you don't experience.

stephenr 10y ago
> > What we seem to forget all too easily is that the cloud is, in its simplest form, just someone else’s computer. > This isn't even actually true for Infrastructure as a Sevice (IaaS), in which case there's significant intellectual capital that's been invested in the automated provisioning of resources for customers. This is much more true with SaaS products. Does all the "intellectual capital" (read: aws/etc runs a highly complex system made up of various parts that sometimes all just shit themselves when a single low level thing hiccups) make it somehow not someone else's computer? The point of "it's just someone else's computer" isn't that it's just a computer like you or I have, it's that it's not some magical, mythical beast. The greatest achievement of aws/et-al is convincing developers they will need "web scale" infrastructure at the drop of a hat.
- andrewstuart2 10y ago
  What I'm saying is that you can't justifiably say that it's "just someone else's computer" (emphasis mine). Just someone else's computer would be ssh root access to a physical machine in some data center. No tenancy separation, no dynamic allocation, no exposure of failure tolerances (e.g. AWS Availability Zones), etc. To illustrate this point succinctly, there are probably thousands of "somebody else's computers" sitting in a dell warehouse somewhere that are no good to me. The cloud is only useful if there are additional services to help me consume that computer. I'm not saying it doesn't run on somebody else's computer. I could, though, since private clouds are in fact "clouds" that are not someone else's computer. Again, what I'm saying is that it is not a sufficiently specific generalization to accurately describe "cloud." The definition of "cloud" is fundamentally more complex than "just somebody else's computer."

devishard 10y ago

> While events like this are not great, isn't it a little myopic to think the solution is to run it yourself? 20 hours of downtime sucks to be sure, but is your own uptime rate really going to be better than theirs? It sounds very much like you're assuming the answer is no. In my entire 10+ year career I've never had a service I was working on have a 20 hour outage. So, probably, the answer is yes.

Xylakant 10y ago
neither did I. but on average, I'm sceptical if my track record is better than theirs, especially on the more complex installations. I've seen cases where a single downed router chopped of a whole building of business people who - since everything was on network drives - couldn't do anything better than play solitaire a whole business day long.
jjbiotech 10y ago
> In my entire 10+ year career I've never had a service I was working on have a 20 hour outage. So, probably, the answer is yes. How arrogant. I seriously doubt the services you've managed have even served a fraction of the data and customers SalesForce has.
- john_fushi 10y ago
  I think his point is that when comparing running SalesForce on the cloud for every users SalesForce has, vs running SalesForce locally for your users, there will a lot less challenge to keep it running locally than on the cloud.
- bisby 10y ago
  The point isn't to serve the volume that SalesForce has, the point is to serve yourself. If you have experience enough to prevent downtime or to handle it when it happens, then perhaps that is better than risking that to someone else. the 20 hours of downtime was probably BECAUSE of the amount of data/customers to handle. whereas if there is 1 customer, recovering isn't nearly as difficult or time consuming.
- 10y ago

dangerboysteve 10y ago

Totally agree. The title is click bait to start and the entire premise of the article is self-serving bullshit.

spelunker 10y ago

I was thinking the same thing. Just like the article said, the cloud is someone else's computer. So why do you think your computer woudln't have these issues?