4 ms·

> While events like this are not great, isn't it a little myopic to think the solution is to run it yourself? 20 hours of downtime sucks to be sure, but is your

> While events like this are not great, isn't it a little myopic to think the solution is to run it yourself? 20 hours of downtime sucks to be sure, but is your own uptime rate really going to be better than theirs? It sounds very much like you're assuming the answer is no. In my entire 10+ year career I've never had a service I was working on have a 20 hour outage. So, probably, the answer is yes.

Xylakant 10y ago
neither did I. but on average, I'm sceptical if my track record is better than theirs, especially on the more complex installations. I've seen cases where a single downed router chopped of a whole building of business people who - since everything was on network drives - couldn't do anything better than play solitaire a whole business day long.
jjbiotech 10y ago
> In my entire 10+ year career I've never had a service I was working on have a 20 hour outage. So, probably, the answer is yes. How arrogant. I seriously doubt the services you've managed have even served a fraction of the data and customers SalesForce has.
- john_fushi 10y ago
  I think his point is that when comparing running SalesForce on the cloud for every users SalesForce has, vs running SalesForce locally for your users, there will a lot less challenge to keep it running locally than on the cloud.
- bisby 10y ago
  The point isn't to serve the volume that SalesForce has, the point is to serve yourself. If you have experience enough to prevent downtime or to handle it when it happens, then perhaps that is better than risking that to someone else. the 20 hours of downtime was probably BECAUSE of the amount of data/customers to handle. whereas if there is 1 customer, recovering isn't nearly as difficult or time consuming.
- devishard 10y ago
  > I seriously doubt the services you've managed have even served a fraction of the data and customers SalesForce has. So what? If your business is built around a system and that system goes down, would you rather that system serve billions of customers and go down for 20 hours, or serve only you and go down for 20 minutes?
deleted 10y ago
[deleted]
outworlder 10y ago
The answer is, as usual, "it depends". Your homegrown service is unlikely to have a 20 hour outage, by being much simpler. Whereas if you use any cloud solution at all, you become a part of a huge and complex system that is likely serving orders of magnitude more requests than your service. And that can go down by reaching situations that would never be seen in your service. The flip side is, your service is probably peanuts to them. Should you grow using your own tech, you'll hit the same pain points that they have, in their very beginning, probably even during the MVP. They have already taken care of that so you won't have to.
gregmac 10y ago
> In my entire 10+ year career I've never had a service I was working on have a 20 hour outage. Through luck or careful planning and good resources? Once you have enough hardware to handle load, it all really comes down to risk mitigation. Assuming it handles load, your service could probably run on a single server, with a single drive, on a DSL connection for years without any issue. I would say if that happens, it has less to do with your skill as an operator and more to do with the luck of not having a power outage, drive failure or backhoe operator dig up a line. Can your service withstand your office/datacenter burning down? Or your datacenter being shutdown with all assets seized by the SEC (true story)? Or a massive earthquake or flood or hurricane that disrupts the entire city?
- devishard 10y ago
  I won't deny the role of luck, but I find that the more careful I am to keep my systems redundant, flexible, and rapidly deployable, the luckier I am. I did have an issue once where a very large power outage that took out a data center. Total downtime was about 50 minutes, which consisted of: 1. Calling the hosting provider and ascertaining the problem. 2. Pulling the latest code onto the fallback server (from a different hosting provider). 3. Restoring the database from the backup server into a newly-created database server. 4. Changing the configs. 2 and 4 were able to happen while 3 was running. As an aside, this is largely why specialized cloud services like AWS or AppEngine that promote vendor lock-in seem crazy to me. AWS can and does go down, and if your infrastructure is built around their tooling, you can't just provision a new box on a new provider.

Xylakant 10y ago

neither did I. but on average, I'm sceptical if my track record is better than theirs, especially on the more complex installations. I've seen cases where a single downed router chopped of a whole building of business people who - since everything was on network drives - couldn't do anything better than play solitaire a whole business day long.

jjbiotech 10y ago

> In my entire 10+ year career I've never had a service I was working on have a 20 hour outage. So, probably, the answer is yes. How arrogant. I seriously doubt the services you've managed have even served a fraction of the data and customers SalesForce has.

john_fushi 10y ago
I think his point is that when comparing running SalesForce on the cloud for every users SalesForce has, vs running SalesForce locally for your users, there will a lot less challenge to keep it running locally than on the cloud.
bisby 10y ago
The point isn't to serve the volume that SalesForce has, the point is to serve yourself. If you have experience enough to prevent downtime or to handle it when it happens, then perhaps that is better than risking that to someone else. the 20 hours of downtime was probably BECAUSE of the amount of data/customers to handle. whereas if there is 1 customer, recovering isn't nearly as difficult or time consuming.
devishard 10y ago
> I seriously doubt the services you've managed have even served a fraction of the data and customers SalesForce has. So what? If your business is built around a system and that system goes down, would you rather that system serve billions of customers and go down for 20 hours, or serve only you and go down for 20 minutes?

deleted 10y ago

[deleted]

outworlder 10y ago

The answer is, as usual, "it depends". Your homegrown service is unlikely to have a 20 hour outage, by being much simpler. Whereas if you use any cloud solution at all, you become a part of a huge and complex system that is likely serving orders of magnitude more requests than your service. And that can go down by reaching situations that would never be seen in your service. The flip side is, your service is probably peanuts to them. Should you grow using your own tech, you'll hit the same pain points that they have, in their very beginning, probably even during the MVP. They have already taken care of that so you won't have to.

gregmac 10y ago

> In my entire 10+ year career I've never had a service I was working on have a 20 hour outage. Through luck or careful planning and good resources? Once you have enough hardware to handle load, it all really comes down to risk mitigation. Assuming it handles load, your service could probably run on a single server, with a single drive, on a DSL connection for years without any issue. I would say if that happens, it has less to do with your skill as an operator and more to do with the luck of not having a power outage, drive failure or backhoe operator dig up a line. Can your service withstand your office/datacenter burning down? Or your datacenter being shutdown with all assets seized by the SEC (true story)? Or a massive earthquake or flood or hurricane that disrupts the entire city?

devishard 10y ago
I won't deny the role of luck, but I find that the more careful I am to keep my systems redundant, flexible, and rapidly deployable, the luckier I am. I did have an issue once where a very large power outage that took out a data center. Total downtime was about 50 minutes, which consisted of: 1. Calling the hosting provider and ascertaining the problem. 2. Pulling the latest code onto the fallback server (from a different hosting provider). 3. Restoring the database from the backup server into a newly-created database server. 4. Changing the configs. 2 and 4 were able to happen while 3 was running. As an aside, this is largely why specialized cloud services like AWS or AppEngine that promote vendor lock-in seem crazy to me. AWS can and does go down, and if your infrastructure is built around their tooling, you can't just provision a new box on a new provider.