3 ms·

Chris told me he was running a 62 node ElasticSearch cluster, which stores over 100 terabytes of data. No doubt, when you are working at that scale, then AWS is

by tshtf 10y ago

Chris told me he was running a 62 node ElasticSearch cluster, which stores over 100 terabytes of data. No doubt, when you are working at that scale, then AWS is very useful. But you need someone with the incredible skills of Chris Clarke, and that is a rare thing. And specialized. Small startups don’t normally have/need/want a devops guy of that skill level, because the devops needs of small startups tend to be minimal. I call bullshit. Startups have/need/want experienced and skilled devops engineers. AWS launched in 2006: there are available, talented devops engineers who understand it. The devops needs of small startups aren't "minimal". Even if you go with a devops engineer not intimately familiar with AWS, he or she can get up to speed on AWS platform operations just as quickly as learning any other new stack the startup may be using... If you're planning on hiring for the lowest common denominator for devops in your startup, you'll be in trouble.

lisa_henderson 10y ago
What about Heroku or Lambda or Cloud Foundry or all the other companies that claim that startups don't need any devops at all?
- tshtf 10y ago
  Good point. If startups are willing to pay much extra for a solution with vendor lock-in, the can certainly follow that path.
- jacques_chester 10y ago
  As a nitpick: Cloud Foundry isn't a company, it's an opensource project. Lambda isn't, uh, quite as fully featured as Heroku or CF yet. And it ties you very closely to AWS. With Heroku you have a reasonable shot at migrating to Cloud Foundry or vice versa. Disclaimer: I work for Pivotal, we're the major donors of engineering to Cloud Foundry. We sell a commercial distribution (PCF) and have a public service (Pivotal Web Services).
pixelmonkey 10y ago
First of all, Chris is great. I'd know: I hired him. (I'm Parse.ly's CTO.) But I will mention that we were not always on AWS. Indeed, I hired Chris at a moment in our company's life when we were growing a bit too fast and still had a "snowflake server" setup in Rackspace Cloud. Served us well for the first year of revenue, but not so well given our growth rate. One of his first jobs was to move us from snowflake to proper config management. Then we made an 18 month pitstop in colo before adopting AWS. I think what the blog post fails to recognize is that there is "commodity AWS use" and then there's "AWS platform use." For example, at Parse.ly we use Route53, Cloudfront, ELB, EC2, S3, RDS, and EMR. The first 5 are basically commodity services. DNS, CDN, load balancer, VMs, backup. Whether you use Rackspace, DigitalOcean, or your own colo, you'll need to figure something out in each category to deploy a real webapp. (Or, accept the devil's bargain of a PaaS.) The last two services, RDS and EMR, are basically AWS-specific value-adds that saves our team time. We used to run our own Postgres EC2 node, but meh, why bother. Our SQL DB is small, and RDS handles a lot of things for us. We also used to run our own Spark clusters (using spark-ec2), but in June 2015, Spark got EMR support, so again, that saves us time/money. We actually breathed a sigh of relief when that came out, since we were essentially building Spark EMR ourselves with boto. Spot instances and reserved instances are very neat cost optimizations at scale. But complex. We adopted RDS/EMR hesitantly because of the "lock-in" they represent. We could definitely run without them. I don't think I have interest in the other 70 AWS services, mainly because I hate lock-in, I like open source, and I want to Keep Things Simple. (So Chris stays sane!) AWS is definitely more complex than DigitalOcean. But I think running production web apps in the cloud is "simply complex", no matter which provider you use. I also think the OP has a good point, which amounts to, "premature scaling is the root of all evil". Parse.ly may run on 200 EC2 nodes across multiple regions & AZs today, but our first prototype ran in a single 1U rackmounted server I built myself and snuck into a friend's server cage (2009). It's important not to waste time on "devops" when you are still figuring out what the heck your startup is even supposed to be doing. If I wasted time on that in the early days, we simply wouldn't be here.
- zer00eyz 10y ago
  >Parse.ly may run on 200 EC2 nodes across multiple regions DevOps has become "developers doing operations" but thats a misapplication of the term. DevOps started as Developers and Operations working together. If your running this many EC2 nodes, and you don't have an Operations person or team, then your running on the razors edge. >At the time of our conversation I had crashed an AWS instance and we were having trouble fixing it. Among the points I made to Sean was that the problem was specific to AWS. If we’d been in the Rackspace cloud, I simply would have rolled back to yesterday’s image. A problem that would have taken 5 minutes to fix instead dragged on for 2 weeks. Honestly you have already been given the sign post that this could go very badly for you. Imagine the case where "crashed instance" was expensive critical infrastructure. How many hours or days of outage before your business is irreparably harmed?

by tshtf 10y ago

lisa_henderson 10y ago

What about Heroku or Lambda or Cloud Foundry or all the other companies that claim that startups don't need any devops at all?

tshtf 10y ago
Good point. If startups are willing to pay much extra for a solution with vendor lock-in, the can certainly follow that path.
jacques_chester 10y ago
As a nitpick: Cloud Foundry isn't a company, it's an opensource project. Lambda isn't, uh, quite as fully featured as Heroku or CF yet. And it ties you very closely to AWS. With Heroku you have a reasonable shot at migrating to Cloud Foundry or vice versa. Disclaimer: I work for Pivotal, we're the major donors of engineering to Cloud Foundry. We sell a commercial distribution (PCF) and have a public service (Pivotal Web Services).

pixelmonkey 10y ago

First of all, Chris is great. I'd know: I hired him. (I'm Parse.ly's CTO.) But I will mention that we were not always on AWS. Indeed, I hired Chris at a moment in our company's life when we were growing a bit too fast and still had a "snowflake server" setup in Rackspace Cloud. Served us well for the first year of revenue, but not so well given our growth rate. One of his first jobs was to move us from snowflake to proper config management. Then we made an 18 month pitstop in colo before adopting AWS. I think what the blog post fails to recognize is that there is "commodity AWS use" and then there's "AWS platform use." For example, at Parse.ly we use Route53, Cloudfront, ELB, EC2, S3, RDS, and EMR. The first 5 are basically commodity services. DNS, CDN, load balancer, VMs, backup. Whether you use Rackspace, DigitalOcean, or your own colo, you'll need to figure something out in each category to deploy a real webapp. (Or, accept the devil's bargain of a PaaS.) The last two services, RDS and EMR, are basically AWS-specific value-adds that saves our team time. We used to run our own Postgres EC2 node, but meh, why bother. Our SQL DB is small, and RDS handles a lot of things for us. We also used to run our own Spark clusters (using spark-ec2), but in June 2015, Spark got EMR support, so again, that saves us time/money. We actually breathed a sigh of relief when that came out, since we were essentially building Spark EMR ourselves with boto. Spot instances and reserved instances are very neat cost optimizations at scale. But complex. We adopted RDS/EMR hesitantly because of the "lock-in" they represent. We could definitely run without them. I don't think I have interest in the other 70 AWS services, mainly because I hate lock-in, I like open source, and I want to Keep Things Simple. (So Chris stays sane!) AWS is definitely more complex than DigitalOcean. But I think running production web apps in the cloud is "simply complex", no matter which provider you use. I also think the OP has a good point, which amounts to, "premature scaling is the root of all evil". Parse.ly may run on 200 EC2 nodes across multiple regions & AZs today, but our first prototype ran in a single 1U rackmounted server I built myself and snuck into a friend's server cage (2009). It's important not to waste time on "devops" when you are still figuring out what the heck your startup is even supposed to be doing. If I wasted time on that in the early days, we simply wouldn't be here.

zer00eyz 10y ago
>Parse.ly may run on 200 EC2 nodes across multiple regions DevOps has become "developers doing operations" but thats a misapplication of the term. DevOps started as Developers and Operations working together. If your running this many EC2 nodes, and you don't have an Operations person or team, then your running on the razors edge. >At the time of our conversation I had crashed an AWS instance and we were having trouble fixing it. Among the points I made to Sean was that the problem was specific to AWS. If we’d been in the Rackspace cloud, I simply would have rolled back to yesterday’s image. A problem that would have taken 5 minutes to fix instead dragged on for 2 weeks. Honestly you have already been given the sign post that this could go very badly for you. Imagine the case where "crashed instance" was expensive critical infrastructure. How many hours or days of outage before your business is irreparably harmed?