13 ms·

How and Why Swiftype Moved from EC2 to Real Hardware

252 points by quicksilver03 12y ago

kovyrin 12y ago
Author here. Happy to answer any questions.
- chrisbolt 12y ago
  What size instances were you using in EC2 that were having performance degradation, and what kind of specs did the real hardware have that you moved to?
- theg2 12y ago
  Using EC2 here for nearly 2 years and you mention I/O problems and instance outages 2-3 times a week. Which size instances were you running? I ask because other than the VM security updates, none our instances have these sort of issues and some of them have a VERY long life (not ideal we know). I understand the cost savings and the rest of the reasoning but in my experience EC2 isn't THAT unreliable.
  - kovyrin 12y ago
    Oh, I know what you're talking about. We too had some instances (actually, a lot of those) that would run for a year with no issues. The problems started around the time you tried to push EC2 instances beyond an "idle, handling some requests just to keep from falling asleep" state. Pushing IO (even with provisioned IOPS) caused random IO stalls, pushing CPU caused REALLY uneven performance, etc. And the only solution provided by EC2 support was always to buy more instances to keep them cold and happy. The problems with that approach (just to name a few): the cost (for a young startup burning money on idle infrastructure like that is not very wise IMO) and the fact, that the time to design, develop and deploy scale-out approach for each of your backend services is the time you could have spent trying to build your product (again, startup-specific; you'll have to think about across-the-board 100% scalability at some point).
    - micahb37 12y ago
      First: I work in Startup BD at AWS (disclosure), but have been a multi-time founder as well. I was under the impression that an AWS architect will sit with you to optimize your infrastructure (Business Support). Did that not happen / or was it not useful? Happy to help in any way I can.
- ryanfitz 12y ago
  Other than ec2, what AWS services were you using and how did you migrate those? At minimum, I'd guess you were using elb for load balancing, sqs for queues and elastic cache for redis.
  - kovyrin 12y ago
    We are still a loyal customer for some of their services. For example, we still use S3 for off-site backups and Route53 is still our primary DNS provider. For load balancing we have moved to a Route53 (health checks and round-robin) + a group of nginx+haproxy+lua-based frontend boxes. Everything else was either built in-house or used open-source components and wasn't really tied to EC2 infrastructure.
- alimoeeny 12y ago
  If you can provide more details on the kind of EC2 instances you were using and the kind of issues that you consider issues that would be great. In my limited experience, I've rarely faced any significant issues with EC2. I assume your threshold for issues must have been very lower than mine.
- samstave 12y ago
  Can you give morespecifics about what you were running on and what you purchased for your own gear? I run an environment that scales to around 1,000 EC2 instances daily. Primarily we run C3.2Xlarge and R3.2xlarge for the core of our application. We have ~12 nodes in our mongo cluster, and havent had a single issue with these nodes. I occasionally get a zombie (totally hung VM) but thats very infrequent. I was aggressively using spot instances previously, but have switched to all 12-month reservations (We would lose many machines to a spot outage, new machines - more than those on Richess) and the recovery time for our system is 35 minutes (due to the R3 boxes needing to download their in-memory index from other machines) - so our service is degraded in capacity until the relaunch of these machines completes. [aside: if youre looking to use spot, do two things - over-provision by a factor of 1.8 and spread across zones, and go look into using ClusterK.com for their balancer product] Anyway, Just curious what was causing "sometimes daily" outages - I can't imagine that this would be due to AWS and not lacking ability of your application to handle instance losses.
- logn 12y ago
  Did you seriously consider a traditional colo or other vendors like SoftLayer (e.g., Rackspace)? It seems like at some point your reasoning here will apply to a colo if you grow bigger.
  - kovyrin 12y ago
    Colo - that's an option I'll try to stay away from as long as it is humanly possible. All of my experiences with colo hardware caused a lifetime of pain so that I'm happy to be paying SL a premium for their outstanding services (I'm a huge fan of Softlayer as you have probably guessed). Re: Rackspace and other providers – based on my real-life practical experience with a few of the largest providers in the States, SL quality of services and their provisioning speed are miles away from competitors could offer. So it was a no-brainer to go with SL and I'm happy we did.
- adevine 12y ago
  How do you handle spikes in request volume? That is, one of the nice things about working in the cloud with dynamic sizing is that your costs should only be relative to your average load, not necessarily your peak load. Given the size of Swiftype, and the fact that you back tons of individual sites (instead of being your own site), there might be enough variance in the sites you back so that your peak and average load is pretty similar. For a single ecommerce site, though, they might get a huge peak over baseline if they do a big marketing push, for example. In that case, they might have to provision many more physical servers than they usually need to handle that peak. Just wanted to see if this issue came up in your planning.
  - kovyrin 12y ago
    Honestly, just as it is with many SAAS companies at some scale, we do not need care about any specific customer's traffic anymore. Simply because we get so much traffic already from our existing customers, that none of the new customers could generate enough to cause any significant blip on the radar. If a customer comes to us with some specific requirements (like being able to index 100MM documents with some specific response time guarantees), we build dedicated pieces of infrastructure for them, load-test it all and provide those guarantees. All of the others are placed in their own pools which have enough capacity to handle 3-5x of ALL of our current traffic with no issues, so any single customer would not be able to generate enough load to cause problems. And, as I mentioned in the article, we could always order new boxes for any of our clusters and get them online within a couple hours, so we are able to scale up pretty quickly if needed.
- risaacs99 12y ago
  Did you reserve your instances? Were you using current generation instances (c3, m3, etc). Did you try to take advantage of traffic patterns to scale up and down the number of instances you were running?
  - kovyrin 12y ago
    We had reserved instanced and regular ones, we did not see any patterns in stability issues between those. Re: instance types - I do not really remember which instances we were using to be honest. And as for the scaling up and down - we had a hard time keeping it all up as it was, we did not want to spend resources trying to make it work with constantly changing node pools (though I understand, that it would push us to building are more robust infrastructure able to handle random node outages, we had a business to build and wanted to focus on the product instead of creating a perfectly scalable application for an early stage application).
    - risaacs99 12y ago
      Well I ask because reserved instances can significantly reduce the price of ec2 (up to something like 75%). Also just turning off idle instances can save a ton. If you invest the time in doing those things I think you can beat other options and so in that way, cloud infrastructure can be very economical. It is like many other things involved in running a technology company. Investing in automation can pay off hugely. The newer instance types are very reliable too (in my experience).
    - count 12y ago
      Did you try dedicated instances?
    - sparkiesaw 12y ago
      If you were using old generation instances (t1 m1 c1 etc.) your experience would have been VERY different to current generation instances (c3 c4 m3 t2). Did you try network optimized instances with (very) low latency networking?
rynop 12y ago
What specific piece saw the biggest boost? My guess is MongoDB. Also if new servers take 1-2 hours, you are always paying for what you "think" will be capacity for peak load correct? How do you handle events that quickly and drastically increase load or txn/sec?
- kovyrin 12y ago
  Actually, it depends. Stability and performance wise, I'd say our Lucene-based search layer has seen the most impressive jump. But yeah, Mongo instances loved the new fast IO as well :-)
mattbeckman 12y ago
"With Amazon we experienced networking issues, hanging VM instances, unpredictable performance degradation (probably due to noisy neighbors sharing our hardware, but there was no way to know) and numerous other problems. " Why do I get the feeling it was kind of a cop-out to just pack up and move without finding the root cause? I've seen it plenty of times: the "best solution" is to just find a different hosting provider. In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code.
- toomuchtodo 12y ago
  > In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code. You haven't been using Amazon long enough then. Amazon is great for proof of concept. No upfront costs, extremely scalable, etc. Unfortunately, its expensive compared to physical hardware once you get to scale, and you may never solve underlying performance issues due to it being a shared tenant environment, even if you're a Netflix-sized customer.
  - josh2600 12y ago
    You can use abstraction layers to isolate yourself from issues with the underlying metal. For example, I had a good thread the last time the maintenance reboots happened: https://news.ycombinator.com/item?id=9120289 https://news.ycombinator.com/item?id=9120289. Solving multi-tenancy issues is hard, but not impossible. I think it's a lot easier with live migration. If a box is giving you problems, just move the load to a new box while maintaining the same IP addressing. With respect to cost, yes, AWS gets expensive at scale, but if you're at scale your servers are generally not your major cost center (it's usually payroll and licensing).
    - deitcher 12y ago
      That is what I like about the cloud. Running in your own hardware lets you be, well, "lazy" about application architecture. Running in a place where it is shared and instances can disappear forces you to design a lot more robustly and nimbly. Of course, that design discipline is great wherever you are running....
nanoGeek 12y ago
One question. If real hardware was always 50% cheaper then AWS wouldn't have been such a success. Can you please explain in which scenarios it makes economical sense to use AWS compared to real hardware?
- jarjoura 12y ago
  AWS is always going to include a premium because they take care of the DevOps portion of your infrastructure. There are plenty of virtual hosting companies that cost significantly less than dedicated hardware, if you won't need all the bells and whistles.
  - robteix 12y ago
    > AWS is always going to include a premium because they take care of the DevOps portion of your infrastructure. But wouldn't that apply also to SoftLayer?
    - hhw 12y ago
      No, there is significantly more complexity, overhead, and R&D to providing cloud services in comparison to bare metal. SoftLayer is actually a very expensive bare metal server provider. There are several good options that cost less than 1/3rd the price. Realistically, at just modest scale (a few physical servers), you should see 1/6th the cost of Amazon. The main benefits of Amazon is that it: a) allows you to scale down i.e. buy services in smaller portions than complete physical servers and b) APIs c) integrated features You could probably pay for one devops position once your infrastructure gets to 10 physical servers.
      - arjunnarayan 12y ago
        Could you please list some competitors to SoftLayer? It's hard to get reliable opinions on cloud providers backed up with actual experience. I'd really appreciate it!
        12y ago
fsniper 12y ago
Oh bare metal reloaded :)
gtrubetskoy 12y ago
This is a no-brainer if you've ever done anything at scale. The explanation is rather simple - hardware is always "on the premise", yours or Amazon's. Someone needs to swap drives, motherboards, man the networking gear, run cables, etc. Amazon doesn't really get a break on the hardware cost because 10,000 servers do not cost less per server than 100 servers (in fact they cost more as the volume goes up if you need them to be identical). When it comes to labor cost - if you have enough hardware for at least one full time datacenter tech, you're in the same boat as Amazon. So you're paying Amazon to do the same work you would do otherwise - only you're subject to their rules and procedures and Amazon being a profitable business needs to mark their services up.
- gaadd33 12y ago
  Why would 10,000 servers cost more than 100 servers? It seems like if you are buying all of your parts in bulk they are going to be cheaper. I'm pretty sure Intel's pricing on CPUs are cheaper by the tray than individually. I know at least when I've bought 20-30 servers at a time, I was able to get a lower cost than when I've only been buying one.
  - sz4kerto 12y ago
    Prices go down until to a certain amount, then then start to increase. Analogy: you want to buy shares of company X. If you buy 1, brokerage costs are high compared to your investment. If you buy 100, you still get the shares from the top of the order book, and fees become negligible. Buy 10M, and you will pay much-much more per share because supply is not going to be there. Just think of the simple supply-demand curve. As demand increases, price increases as well. The bulk discount pricing is only valid for amounts that provide better utilization of the supply chain. If Intel can produce 1M chips a month, then if somebody orders the last 50k, he might get a discount. If someone wants 2M, then he needs to pay a huge markup because the supply chain is not ready. And Amazon is definitely big enough to move the equilibrium price up.
    - Someone1234 12y ago
      I see no explanation for why suppliers wouldn't match demand. Aside from the HDD shortage which hit everyone, I've seen no issues like you're describing where essentially the market runs out of servers/CPUs/etc. Because that's the only way this theory applies, if they're completely unable to meet the demand due to some specific shortage in the market. The only reason your shares analogy makes sense is because there ARE a finite number of shares available at any given time, and buying too many drives up the cost in the entire market. Most manufacturers can scale up production as demand increases.
davidjgraph 12y ago
It strikes me that more and more a critical selection when growing in this way is the DNS part. It needs to be back-end agnostic and provide an increasing amount of functionality. Health checks and failover are must have now, but this article makes me wonder three things: 1) Are there any DNS services that understand geography of your "zones", i.e. route to and failover based on IP? (but are still platform agnostic). 2) How long can a DNS failover take worst case? You can technically set a low TTL, but don't a lot of ISPs just increase that to a minimum? 3) Isn't it better to replace some of the DNS failover with high availability dedicated load balancing?
- kovyrin 12y ago
  We use DNS-based load balancing along with an HA pair of load balancers in each datacenter. If the DNS health check fails, we stop sending traffic to a failing frontend LB. If failing LB is dead, we move its IP to the other one. DNS TTL is not as big of an issue today as it was 5-10 years ago, when idiotic ISPs were trying to save on DNS resolving by ignoring TTLs. Nowadays you see an almost perfect drop in traffic when switching off a load balancer. Only bots and some weird exotic ISPs may keep sending traffic to a disabled box for up to an hour or two, but since DNS LB is only used to handle real emergency outages and for planned maintenance we could move LB IPs around, I really do not see it as a big enough issue to stop using the DNS LB magic :-)
- hhw 12y ago
  1) Yes, there are several DNS service providers that offer BGP anycast with geographically aware failover / load balancing. UltraDNS and DYN are the larger ones. 2) Yes, some ISPs do set a minimum TTL. Although BGP anycast is the most effective as the first line, sometimes it makes sense to have your reverse proxy caching layer override that distribution based on GeoIP and redirect to a more suitable proxy node closer to the client. This is especially the case when people using recursive lookup DNS servers that aren't necessarily geographically close to them (e.g. 8.8.8.8). It could also be useful in cases where TTL expiration hasn't caught up yet though. 3) No. Think of BGP Anycast DNS as distribution at a global level, and dedicated load balancers as distribution at the local level. You need to work out how to get the traffic to the load balancer first, and load balancing across distant geographies (high latency) results in horrible performance.
bobofettfett 12y ago
My 5cent: If you have lots and lots of money and a high margin business, do yourself a favor and go with Amazon (much less hassle with contract management and low level challenges). If you need to scale month to month and are growing 50% per month, go with Amazon. If you are very small and can live with 10 instances, go with Amazon. If CAPEX doesn't help you and for whatever reasons you need to spend OPEX, go with Amazon. If you need many (types of) machines for failover but which otherwise mostly idle, go with Amazon. Otherwise it's always cheaper to buy or rent hardware. Amazon is very expensive (TCO). If you base your decision on hype, you're screwed. * Amazon stands for Cloud Provider, personally I'm choosing Digital Ocean with Mesos/Docker. * Except S3 which is a no brainer to use.
- kovyrin 12y ago
  Absolutely, 100% agree. Thanks for the comment!
- chx 12y ago
  > If you need to scale month to month and are growing 50% per month, Then rent more servers. > If you are very small and can live with 10 instances, Then rent a few servers. I maintain that there are extremely few cases for a typical website to use the cloud. To handle peaks, it is both simpler and cheaper to keep enough capacity just idling around than spinning up and down Amazon instances. The cloud is almost always a useless hype. It can be different if you can architect to use the various services Amazon provides.
  - AstroChimpHam 12y ago
    Why rent more servers when it's so much easier to spin them up as you need them in AWS and tear them down (and no longer pay for them) when you don't. If you're expecting unknown change in scale, it's so much easier to be able to just spin up servers to keep up than anything else. > To handle peaks, it is both simpler and cheaper to keep enough capacity just idling around than spinning up and down Amazon instances. At a growing website, you have no idea how much "enough" is. Why try to estimate caps when you don't have to?
  - 12y ago
snoopybbt 12y ago
There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: Cloud computing offers you the great and awesome advantages of being able to instantly scale your application, replicate your data and basically just grow according to your business volume, and all this without significant investments, delivery time, setup time, people time, maintenance or anything but it's expensive in the long run. And this is OKAY, this is GREAT. Once you're big enough, you know what your load is now and what your load will likely be, and you know exactly what you need now and (approximately) what you're going to need in the near future, setting up your own datacenter is way, way more effective. Amazon does not get free electricity, free servers and/or free people time. Of course, you're paying that, and you're also paying Amazon's profits. This is absolutely fine, as long their service fits you. But when you grow enough, put simply, your needs change. It's just that.
- rodgerd 12y ago
  > There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: I wish it were true, but plenty of companies are gripped by cloud fever. I've seen quite a few going down the route of charging into the cloud not because they've run the numbers and found it stacks up, but because they want to be in the cloud, and Amazon have some great marketing people.
  - yeukhon 12y ago
    From personal experience this is absolutely true. The initial cost to migrate and learn the cloud practice is unbelievable, but once you have the process in place running on Amazon CAN save some cost down the road, including the hours you need to replace hardware. You will build tools or use existing tools to create your infrastructure and operation process. A generous estimation to reach a level of maturity is 1.5 year. For ever-growing business this cloud fever is acceptable. You can definitely save cost by subscribing to reserved instances, but the downside is you have to put down money upfront, which is very hard for many small players out there. But watch out if you run data pipeline jobs - sometimes your so-called big data is really not that big. A few GBs daily report doesn't need to run on c3.xlarge instances. They can do just fine with a 24/7 m3.large instance. There was an article on HN a while ago about how one could run a custom report with shell commands on a commodity hardware, and get 100x times performance compare to running on EMR. You can also consider running most of your jobs on premise. Ihe network banwidth in/out is probably going to be cheaper than running all of your jobs on EMR. Direct connect is a great choice to boost the connectivity stability and security. Go for it. Cloud is great for HA, because on Amazon you are encourage to build in multi-AZ and even multi-region. S3 is absolutely the de-facto today IMO for object storage. It's cheap and reliable. The learning curve for proper Amazon (or just about any cloud provider) is really deep. You can either end up like running Black Friday sales, or running like Netflix with monkey enjoying tea. Running on cloud is no different than running on-premise, just you have to start all over again, because now you have to re-consider network, security, monitoring, and practice.
anonbanker 12y ago
it's about a 5-7 year cycle; IT services are outsourced (this time, to the cloud), but as time progresses, all of those services are consolidated (usually, under new management) in-house. I used to be able to set a watch by HP's cycle before the EDS buyout.
forgottenacc56 12y ago
I'm telling you theres real opportunity for someone to create "the Uber of colocation/dedicated hosting".
- superuser2 12y ago
  OpenStack was supposed to facillitate this. Doesn't seem to get much use.
  - AndyNemmity 12y ago
    It does from my perspective. I could list many companies that use it for major things. It doesn't always get publicity. I work for a major company, and we don't scream from the roof tops about it.
anthony_barker 12y ago
In my experience firewalls and network is more expensive than servers.... Anyone offer competition to VPC?
- kovyrin 12y ago
  Softlayer manages the network for us. They offer a fully isolated backend network (frontend connection to the world is optional). For frontend connections we simply use iptables.
shimon_e 12y ago
RunAbove from OVH seems to be the best of both worlds. You can rent dedicated hardware by the hour for very reasonable prices.
- kovyrin 12y ago
  AFAIK Softlayer has hourly-based rental for real hardware as well. Never tried that, so could not comment on it though.
johnnycarcin 12y ago
Interesting article, thanks for sharing. We actually just went the exact opposite because of the larger scale issues we were having with Softlayer. Do you feel like you lost any resiliency by making the switch to physical servers (more virtual instances on one physical server, servers in the same rack, etc)?
- kovyrin 12y ago
  No, I really do not think going to EC2 could be beneficial in any way in terms of improving resiliency compared to Softlayer. SL allows you to control which VLANs your box will end up on. VLANs could be treated as racks (since they do not allocate more than one VLAN per rack). Then you have multiple DCs in one region (e.g. DAL01, DAL05, DAL07, etc) and you have many different regions (DAL, SEA, WAS, AMS, etc). I'd be very interested what problems you were having with them and at what scale. If this is a private topic, we could do it over email or some other medium if you like. You can contact me by any of the means listed here: http://kovyrin.net/contact/ http://kovyrin.net/contact/
  - johnnycarcin 12y ago
    We were about 75% virtual with SL and 25% bare metal. One of our issues with the virtual stuff is when we started dedicating them to a set VLAN, multiple times we ran into an issue where some type of resource for that pod the VLAN was in would be maxed out (usually storage) so we couldn't create a new instance. The solution we were given was to let the system pick a VLAN but by doing that we had lost control of the placement and added some complexity to our architecture. Aside from that it was mainly nit-picky type stuff, but still things that were annoying (networking issues between DCs, networking issues between pods, internal mirrored apt-get repos going out of sync, API is kind of blah, etc). We use docker so having a few bare metal machines with tons of containers on them wasn't a great HA setup (for us at least), even running in two data centers. The fairly quick setup time though was a nice selling point. When we went to AWS things just kind of worked. The API was easier to use and the GUI portal was way nicer/stable. So far we have not had any odd issues with our instances, but we also typically run them at about 50% capacity so that might be why. It is also still early so maybe things will come up in 6+ months that send us back to SL :)
peterwwillis 12y ago
Cases where people try to use EC2: - We're starting from scratch and think AWS will give us flexibility for cheap - We have existing servers and think moving from them to AWS will give us flexibility for cheap Cases where people move away from EC2: - It was slow/unreliable - It was expensive Conlusion: You should use AWS EC2 in order to save money and have flexible resource allocation, but don't expect it to be stable or cost-effective.
tschellenbach 12y ago
From my experience AWS is one of the most reliable hosting providers. It's extremely easy to setup a fault tolerant infrastructure using cloudformation, puppet/chef etc, boto and a bit of autoscaling. The only disadvantage AWS has is costs. In many cases it's more expensive than a traditional hosting provider. On the flip side, your engineers will be more productive. What I miss in this article is any details on why they had issues with AWS. You can't just say it wasn't reliable and not explain the details. AWS works for all of the world's largest startups, why didn't it work for Swiftype?
stephen-mw 12y ago
Here's my anecdotal, one-data-point experience from moving a giant EC2 environment to datacenter: 1. Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. 2. Any weirdness you experienced with AWS infrastructure will be replaced with weirdness in your own environment, except now you're on the line to troubleshoot and fix it yourself. 3. Operation staff will immediately start guarding the food bowl as resources become finite. Server provision waits start to seem like breadlines. Power is consolidated with Those Whom You Must Ask. 4. Your cost will decrease, sometimes significantly so. 5. You'll have more hardware flexibility to run your app just the way you want to (Stack Overflow's mega databases come to mind[0]). In the end I think this type of transition is for stable companies that don't mind or even prefer strong divisions of labor (coders who code, sysadmins who sysadmin, testers who test), but it's not for startups or companies that hope to move with any kind of strong velocity. [0] http://highscalability.com/blog/2014/7/21/stackoverflow-update-560m-pageviews-a-month-25-servers-and-i.html http://highscalability.com/blog/2014/7/21/stackoverflow-upda...
- chx 12y ago
  > Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. Why? What are you talking about? You are hiring servers you are no colo'ing them. Networking them is not your problem. Your responsibility still starts from a root prompt just there's no VM layer between that and the physical server.
  - kovyrin 12y ago
    Yes, it really does look like @stephen-mw is talking about a migration to an owned/colocated hardware. And that's a very different kind of hairy mess I'd prefer to stay away from as long as humanly possible.
    - chx 12y ago
      OMG, colo. I've been the architect of a Top 500 website acquired by a Top 100 and even for that we just rented a few servers (you'd be surprised how few -- a dozen or so). I do not really know how big you need to be make colo a sensible choice but quite probably much, much bigger than most think of. We ran the numbers and they weren't pretty. And for cloud: we purchased video conversion as a service, that one ran in the cloud. I can see how that makes sense.
deitcher 12y ago
Oleksiy, can you share the economics? You said 50% savings. What was everything you had running in EC2 (easy to figure out the costs), what were your equivalents in softlayer? Would be very interesting to see the economics.
icedchai 12y ago
I'm convinced hybrid cloud is the way to go. Anything needing high IO performance should be on dedicated. Anything needed CPU/memory elasticity (worker nodes, etc) should be in the cloud. Assuming you can get low latency connectivity into AWS with DirectConnect, this might work?
- kovyrin 12y ago
  Yeah, that (worker nodes, async processing, etc) seems like one of those ideal use-case for clouds that I could definitely support as a viable option for companies that aren't ready for all-in cloud deployment.
- fleitz 12y ago
  Cpu also generally sucks due to cache thrashing on non-trivial datasets
  - icedchai 12y ago
    Heh. So what is a good use case for EC2?
    - fleitz 12y ago
      When you have highly variable load. eg. Netflix probably spins up thousands of servers for a few hours.
- nasalgoat 12y ago
  The speed of light is not your friend. You can never compete with the latency of a local network versus even the fastest connection to a cloud provider.
amazon_not 12y ago
The biggest problem with AWS is the outrageous cost of bandwidth. Even if you ignore all the other cost differentials, the bandwidth charges will kill you at scale. Unfortunately cloud computing, or at the very least AWS, overpromises and underdelivers at scale. All in all economically viable use cases for cloud computing are very few and very specific at scale.
- wmf 12y ago
  Since people are sharing their experiences in this thread, has anyone tried using Direct Connect to get cheaper bandwidth?
  - amazon_not 12y ago
    There aren't a lot of savings to be had. In addition to paying for the Direct Connect infrastructure you still have to pay Amazon per GB charges, albeit at a slightly lesser rate. The only real solution is to move the bandwidth usage off AWS.
qaq 12y ago
Finally sane take on Cloud BS :)
hashtree 12y ago
A concrete example of how I saved a few hundred thousand dollars over the AWS by building quarter rack colocation setups with SuperMicro servers: https://gist.github.com/rockymadden/5561377 https://gist.github.com/rockymadden/5561377 With Ansible, I spend no more than an hour a week, amortized, maintaining both the hardware and administration. I assume nodes for any specific role will fail, I only scale horizontally, I always have redundancy for every role, I stay off disk as long as possible (heya 512GB RAM redis cluster), etc.
- danieltillett 12y ago
  Great resource. Are you going to keep this updated?
abalone 12y ago
"2-3 serious outages a week" with EC2?? Details, please.
- kovyrin 12y ago
  Those weren't outages for a specific instance. But from the whole pool of instances we were running 2-3 would have networking issues, random unexplained hangs requiring an instance restart, huge CPU performance drops, IO hangs, etc, etc.
billyhoffman 12y ago
I used Rackspace (and later EC2) originally because of upfront capital costs. I bootstrapped my company, and paying a thousand dollars or more, plus hosting/racking fees, just wasn't doable. $50-100 bucks for a reasonable Windows server? Easy deal to make when cash flow is small.
jonpress 12y ago
I think cloud providers are mostly good for small/medium, fast growing startups. Big companies need more granular flexibility. The path to vertical integration in the software indsutry is a lot smoother than in many other industries, there are many levels: You can... 1. Make your own hardware 2. Own/manage your own hardware 3. Rent commodity hardware from a standard hosting provider 4. Use IaaS (e.g. EC2) 5. Use PaaS (e.g. AppFog, Nodejitsu) 6. Use BaaS (e.g. Firebase, PubNub, Pusher.com) The higher the level, the more technical flexibility you lose. The bigger the company, the more it makes sense operate at a lower level because there is no significant wastage being introduced as you move down the levels (you remove the middlemen so you can pocket their profits) and the capital cost to move between levels is relatively low. If you compare software to another industry like cheesemaking for example, if you're a cheesemaker and you want to make your own milk, the next step is to buy the whole farm and then you have to figure out what do do with the meat (wastage). Going between these two levels is expensive and could mean doubling or tripling your expenses so it's not an easy move to make.
deleted 12y ago
[deleted]
jamiesonbecker 12y ago
Cost-wise, AWS makes a lot of sense when growth is not easy to project, happens rapidly, or varies (seasonally, randomly, etc). Of course, some of this has to do with your team's skill level, but I've had clients run up $100k+ monthly bills at AWS with a relatively small build-out. (and, wow, VPC migrations..) For fixed or predictable growth patterns on a mature app/platform, a slow build-out on real iron will generally be significantly less expensive, all other things being equal. However, there are other advantages to AWS that gets lost in this story, such as pre-built, highly scalable datastores. Comparing EC2 to real iron misses most of the real story on why the cloud is changing everything. One of the hardest things I have to tell clients is not to build their own datastore/database in house or at EC2; sometimes the case is clearcut, and sometimes not so much, but if you have a datastore at AWS that gives you 80% of what you need, use it instead of rolling your own. (source: IAMA AWS Growth Architect)
BackOel 12y ago
Thanks for the information

252 points by quicksilver03 12y ago

kovyrin 12y ago

Author here. Happy to answer any questions.

chrisbolt 12y ago
What size instances were you using in EC2 that were having performance degradation, and what kind of specs did the real hardware have that you moved to?
theg2 12y ago
Using EC2 here for nearly 2 years and you mention I/O problems and instance outages 2-3 times a week. Which size instances were you running? I ask because other than the VM security updates, none our instances have these sort of issues and some of them have a VERY long life (not ideal we know). I understand the cost savings and the rest of the reasoning but in my experience EC2 isn't THAT unreliable.
- kovyrin 12y ago
  Oh, I know what you're talking about. We too had some instances (actually, a lot of those) that would run for a year with no issues. The problems started around the time you tried to push EC2 instances beyond an "idle, handling some requests just to keep from falling asleep" state. Pushing IO (even with provisioned IOPS) caused random IO stalls, pushing CPU caused REALLY uneven performance, etc. And the only solution provided by EC2 support was always to buy more instances to keep them cold and happy. The problems with that approach (just to name a few): the cost (for a young startup burning money on idle infrastructure like that is not very wise IMO) and the fact, that the time to design, develop and deploy scale-out approach for each of your backend services is the time you could have spent trying to build your product (again, startup-specific; you'll have to think about across-the-board 100% scalability at some point).
  - micahb37 12y ago
    First: I work in Startup BD at AWS (disclosure), but have been a multi-time founder as well. I was under the impression that an AWS architect will sit with you to optimize your infrastructure (Business Support). Did that not happen / or was it not useful? Happy to help in any way I can.
ryanfitz 12y ago
Other than ec2, what AWS services were you using and how did you migrate those? At minimum, I'd guess you were using elb for load balancing, sqs for queues and elastic cache for redis.
- kovyrin 12y ago
  We are still a loyal customer for some of their services. For example, we still use S3 for off-site backups and Route53 is still our primary DNS provider. For load balancing we have moved to a Route53 (health checks and round-robin) + a group of nginx+haproxy+lua-based frontend boxes. Everything else was either built in-house or used open-source components and wasn't really tied to EC2 infrastructure.
alimoeeny 12y ago
If you can provide more details on the kind of EC2 instances you were using and the kind of issues that you consider issues that would be great. In my limited experience, I've rarely faced any significant issues with EC2. I assume your threshold for issues must have been very lower than mine.
samstave 12y ago
Can you give morespecifics about what you were running on and what you purchased for your own gear? I run an environment that scales to around 1,000 EC2 instances daily. Primarily we run C3.2Xlarge and R3.2xlarge for the core of our application. We have ~12 nodes in our mongo cluster, and havent had a single issue with these nodes. I occasionally get a zombie (totally hung VM) but thats very infrequent. I was aggressively using spot instances previously, but have switched to all 12-month reservations (We would lose many machines to a spot outage, new machines - more than those on Richess) and the recovery time for our system is 35 minutes (due to the R3 boxes needing to download their in-memory index from other machines) - so our service is degraded in capacity until the relaunch of these machines completes. [aside: if youre looking to use spot, do two things - over-provision by a factor of 1.8 and spread across zones, and go look into using ClusterK.com for their balancer product] Anyway, Just curious what was causing "sometimes daily" outages - I can't imagine that this would be due to AWS and not lacking ability of your application to handle instance losses.
logn 12y ago
Did you seriously consider a traditional colo or other vendors like SoftLayer (e.g., Rackspace)? It seems like at some point your reasoning here will apply to a colo if you grow bigger.
- kovyrin 12y ago
  Colo - that's an option I'll try to stay away from as long as it is humanly possible. All of my experiences with colo hardware caused a lifetime of pain so that I'm happy to be paying SL a premium for their outstanding services (I'm a huge fan of Softlayer as you have probably guessed). Re: Rackspace and other providers – based on my real-life practical experience with a few of the largest providers in the States, SL quality of services and their provisioning speed are miles away from competitors could offer. So it was a no-brainer to go with SL and I'm happy we did.
adevine 12y ago
How do you handle spikes in request volume? That is, one of the nice things about working in the cloud with dynamic sizing is that your costs should only be relative to your average load, not necessarily your peak load. Given the size of Swiftype, and the fact that you back tons of individual sites (instead of being your own site), there might be enough variance in the sites you back so that your peak and average load is pretty similar. For a single ecommerce site, though, they might get a huge peak over baseline if they do a big marketing push, for example. In that case, they might have to provision many more physical servers than they usually need to handle that peak. Just wanted to see if this issue came up in your planning.
- kovyrin 12y ago
  Honestly, just as it is with many SAAS companies at some scale, we do not need care about any specific customer's traffic anymore. Simply because we get so much traffic already from our existing customers, that none of the new customers could generate enough to cause any significant blip on the radar. If a customer comes to us with some specific requirements (like being able to index 100MM documents with some specific response time guarantees), we build dedicated pieces of infrastructure for them, load-test it all and provide those guarantees. All of the others are placed in their own pools which have enough capacity to handle 3-5x of ALL of our current traffic with no issues, so any single customer would not be able to generate enough load to cause problems. And, as I mentioned in the article, we could always order new boxes for any of our clusters and get them online within a couple hours, so we are able to scale up pretty quickly if needed.
risaacs99 12y ago
Did you reserve your instances? Were you using current generation instances (c3, m3, etc). Did you try to take advantage of traffic patterns to scale up and down the number of instances you were running?
- kovyrin 12y ago
  We had reserved instanced and regular ones, we did not see any patterns in stability issues between those. Re: instance types - I do not really remember which instances we were using to be honest. And as for the scaling up and down - we had a hard time keeping it all up as it was, we did not want to spend resources trying to make it work with constantly changing node pools (though I understand, that it would push us to building are more robust infrastructure able to handle random node outages, we had a business to build and wanted to focus on the product instead of creating a perfectly scalable application for an early stage application).
  - risaacs99 12y ago
    Well I ask because reserved instances can significantly reduce the price of ec2 (up to something like 75%). Also just turning off idle instances can save a ton. If you invest the time in doing those things I think you can beat other options and so in that way, cloud infrastructure can be very economical. It is like many other things involved in running a technology company. Investing in automation can pay off hugely. The newer instance types are very reliable too (in my experience).
  - count 12y ago
    Did you try dedicated instances?
  - sparkiesaw 12y ago
    If you were using old generation instances (t1 m1 c1 etc.) your experience would have been VERY different to current generation instances (c3 c4 m3 t2). Did you try network optimized instances with (very) low latency networking?

rynop 12y ago

What specific piece saw the biggest boost? My guess is MongoDB. Also if new servers take 1-2 hours, you are always paying for what you "think" will be capacity for peak load correct? How do you handle events that quickly and drastically increase load or txn/sec?

kovyrin 12y ago
Actually, it depends. Stability and performance wise, I'd say our Lucene-based search layer has seen the most impressive jump. But yeah, Mongo instances loved the new fast IO as well :-)

mattbeckman 12y ago

"With Amazon we experienced networking issues, hanging VM instances, unpredictable performance degradation (probably due to noisy neighbors sharing our hardware, but there was no way to know) and numerous other problems. " Why do I get the feeling it was kind of a cop-out to just pack up and move without finding the root cause? I've seen it plenty of times: the "best solution" is to just find a different hosting provider. In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code.

toomuchtodo 12y ago
> In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code. You haven't been using Amazon long enough then. Amazon is great for proof of concept. No upfront costs, extremely scalable, etc. Unfortunately, its expensive compared to physical hardware once you get to scale, and you may never solve underlying performance issues due to it being a shared tenant environment, even if you're a Netflix-sized customer.
- josh2600 12y ago
  You can use abstraction layers to isolate yourself from issues with the underlying metal. For example, I had a good thread the last time the maintenance reboots happened: https://news.ycombinator.com/item?id=9120289 https://news.ycombinator.com/item?id=9120289. Solving multi-tenancy issues is hard, but not impossible. I think it's a lot easier with live migration. If a box is giving you problems, just move the load to a new box while maintaining the same IP addressing. With respect to cost, yes, AWS gets expensive at scale, but if you're at scale your servers are generally not your major cost center (it's usually payroll and licensing).
  - deitcher 12y ago
    That is what I like about the cloud. Running in your own hardware lets you be, well, "lazy" about application architecture. Running in a place where it is shared and instances can disappear forces you to design a lot more robustly and nimbly. Of course, that design discipline is great wherever you are running....

nanoGeek 12y ago

One question. If real hardware was always 50% cheaper then AWS wouldn't have been such a success. Can you please explain in which scenarios it makes economical sense to use AWS compared to real hardware?

jarjoura 12y ago
AWS is always going to include a premium because they take care of the DevOps portion of your infrastructure. There are plenty of virtual hosting companies that cost significantly less than dedicated hardware, if you won't need all the bells and whistles.
- robteix 12y ago
  > AWS is always going to include a premium because they take care of the DevOps portion of your infrastructure. But wouldn't that apply also to SoftLayer?
  - hhw 12y ago
    No, there is significantly more complexity, overhead, and R&D to providing cloud services in comparison to bare metal. SoftLayer is actually a very expensive bare metal server provider. There are several good options that cost less than 1/3rd the price. Realistically, at just modest scale (a few physical servers), you should see 1/6th the cost of Amazon. The main benefits of Amazon is that it: a) allows you to scale down i.e. buy services in smaller portions than complete physical servers and b) APIs c) integrated features You could probably pay for one devops position once your infrastructure gets to 10 physical servers.
    - arjunnarayan 12y ago
      Could you please list some competitors to SoftLayer? It's hard to get reliable opinions on cloud providers backed up with actual experience. I'd really appreciate it!
      - 12y ago

fsniper 12y ago

Oh bare metal reloaded :)

gtrubetskoy 12y ago

This is a no-brainer if you've ever done anything at scale. The explanation is rather simple - hardware is always "on the premise", yours or Amazon's. Someone needs to swap drives, motherboards, man the networking gear, run cables, etc. Amazon doesn't really get a break on the hardware cost because 10,000 servers do not cost less per server than 100 servers (in fact they cost more as the volume goes up if you need them to be identical). When it comes to labor cost - if you have enough hardware for at least one full time datacenter tech, you're in the same boat as Amazon. So you're paying Amazon to do the same work you would do otherwise - only you're subject to their rules and procedures and Amazon being a profitable business needs to mark their services up.

gaadd33 12y ago
Why would 10,000 servers cost more than 100 servers? It seems like if you are buying all of your parts in bulk they are going to be cheaper. I'm pretty sure Intel's pricing on CPUs are cheaper by the tray than individually. I know at least when I've bought 20-30 servers at a time, I was able to get a lower cost than when I've only been buying one.
- sz4kerto 12y ago
  Prices go down until to a certain amount, then then start to increase. Analogy: you want to buy shares of company X. If you buy 1, brokerage costs are high compared to your investment. If you buy 100, you still get the shares from the top of the order book, and fees become negligible. Buy 10M, and you will pay much-much more per share because supply is not going to be there. Just think of the simple supply-demand curve. As demand increases, price increases as well. The bulk discount pricing is only valid for amounts that provide better utilization of the supply chain. If Intel can produce 1M chips a month, then if somebody orders the last 50k, he might get a discount. If someone wants 2M, then he needs to pay a huge markup because the supply chain is not ready. And Amazon is definitely big enough to move the equilibrium price up.
  - Someone1234 12y ago
    I see no explanation for why suppliers wouldn't match demand. Aside from the HDD shortage which hit everyone, I've seen no issues like you're describing where essentially the market runs out of servers/CPUs/etc. Because that's the only way this theory applies, if they're completely unable to meet the demand due to some specific shortage in the market. The only reason your shares analogy makes sense is because there ARE a finite number of shares available at any given time, and buying too many drives up the cost in the entire market. Most manufacturers can scale up production as demand increases.

davidjgraph 12y ago

It strikes me that more and more a critical selection when growing in this way is the DNS part. It needs to be back-end agnostic and provide an increasing amount of functionality. Health checks and failover are must have now, but this article makes me wonder three things: 1) Are there any DNS services that understand geography of your "zones", i.e. route to and failover based on IP? (but are still platform agnostic). 2) How long can a DNS failover take worst case? You can technically set a low TTL, but don't a lot of ISPs just increase that to a minimum? 3) Isn't it better to replace some of the DNS failover with high availability dedicated load balancing?

kovyrin 12y ago
We use DNS-based load balancing along with an HA pair of load balancers in each datacenter. If the DNS health check fails, we stop sending traffic to a failing frontend LB. If failing LB is dead, we move its IP to the other one. DNS TTL is not as big of an issue today as it was 5-10 years ago, when idiotic ISPs were trying to save on DNS resolving by ignoring TTLs. Nowadays you see an almost perfect drop in traffic when switching off a load balancer. Only bots and some weird exotic ISPs may keep sending traffic to a disabled box for up to an hour or two, but since DNS LB is only used to handle real emergency outages and for planned maintenance we could move LB IPs around, I really do not see it as a big enough issue to stop using the DNS LB magic :-)
hhw 12y ago
1) Yes, there are several DNS service providers that offer BGP anycast with geographically aware failover / load balancing. UltraDNS and DYN are the larger ones. 2) Yes, some ISPs do set a minimum TTL. Although BGP anycast is the most effective as the first line, sometimes it makes sense to have your reverse proxy caching layer override that distribution based on GeoIP and redirect to a more suitable proxy node closer to the client. This is especially the case when people using recursive lookup DNS servers that aren't necessarily geographically close to them (e.g. 8.8.8.8). It could also be useful in cases where TTL expiration hasn't caught up yet though. 3) No. Think of BGP Anycast DNS as distribution at a global level, and dedicated load balancers as distribution at the local level. You need to work out how to get the traffic to the load balancer first, and load balancing across distant geographies (high latency) results in horrible performance.

bobofettfett 12y ago

My 5cent: If you have lots and lots of money and a high margin business, do yourself a favor and go with Amazon (much less hassle with contract management and low level challenges). If you need to scale month to month and are growing 50% per month, go with Amazon. If you are very small and can live with 10 instances, go with Amazon. If CAPEX doesn't help you and for whatever reasons you need to spend OPEX, go with Amazon. If you need many (types of) machines for failover but which otherwise mostly idle, go with Amazon. Otherwise it's always cheaper to buy or rent hardware. Amazon is very expensive (TCO). If you base your decision on hype, you're screwed. * Amazon stands for Cloud Provider, personally I'm choosing Digital Ocean with Mesos/Docker. * Except S3 which is a no brainer to use.

kovyrin 12y ago
Absolutely, 100% agree. Thanks for the comment!
chx 12y ago
> If you need to scale month to month and are growing 50% per month, Then rent more servers. > If you are very small and can live with 10 instances, Then rent a few servers. I maintain that there are extremely few cases for a typical website to use the cloud. To handle peaks, it is both simpler and cheaper to keep enough capacity just idling around than spinning up and down Amazon instances. The cloud is almost always a useless hype. It can be different if you can architect to use the various services Amazon provides.
- AstroChimpHam 12y ago
  Why rent more servers when it's so much easier to spin them up as you need them in AWS and tear them down (and no longer pay for them) when you don't. If you're expecting unknown change in scale, it's so much easier to be able to just spin up servers to keep up than anything else. > To handle peaks, it is both simpler and cheaper to keep enough capacity just idling around than spinning up and down Amazon instances. At a growing website, you have no idea how much "enough" is. Why try to estimate caps when you don't have to?
- 12y ago

snoopybbt 12y ago

There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: Cloud computing offers you the great and awesome advantages of being able to instantly scale your application, replicate your data and basically just grow according to your business volume, and all this without significant investments, delivery time, setup time, people time, maintenance or anything but it's expensive in the long run. And this is OKAY, this is GREAT. Once you're big enough, you know what your load is now and what your load will likely be, and you know exactly what you need now and (approximately) what you're going to need in the near future, setting up your own datacenter is way, way more effective. Amazon does not get free electricity, free servers and/or free people time. Of course, you're paying that, and you're also paying Amazon's profits. This is absolutely fine, as long their service fits you. But when you grow enough, put simply, your needs change. It's just that.

rodgerd 12y ago
> There's not much need for a fancy article on a fancy website in order to understand a key concept of cloud computing: I wish it were true, but plenty of companies are gripped by cloud fever. I've seen quite a few going down the route of charging into the cloud not because they've run the numbers and found it stacks up, but because they want to be in the cloud, and Amazon have some great marketing people.
- yeukhon 12y ago
  From personal experience this is absolutely true. The initial cost to migrate and learn the cloud practice is unbelievable, but once you have the process in place running on Amazon CAN save some cost down the road, including the hours you need to replace hardware. You will build tools or use existing tools to create your infrastructure and operation process. A generous estimation to reach a level of maturity is 1.5 year. For ever-growing business this cloud fever is acceptable. You can definitely save cost by subscribing to reserved instances, but the downside is you have to put down money upfront, which is very hard for many small players out there. But watch out if you run data pipeline jobs - sometimes your so-called big data is really not that big. A few GBs daily report doesn't need to run on c3.xlarge instances. They can do just fine with a 24/7 m3.large instance. There was an article on HN a while ago about how one could run a custom report with shell commands on a commodity hardware, and get 100x times performance compare to running on EMR. You can also consider running most of your jobs on premise. Ihe network banwidth in/out is probably going to be cheaper than running all of your jobs on EMR. Direct connect is a great choice to boost the connectivity stability and security. Go for it. Cloud is great for HA, because on Amazon you are encourage to build in multi-AZ and even multi-region. S3 is absolutely the de-facto today IMO for object storage. It's cheap and reliable. The learning curve for proper Amazon (or just about any cloud provider) is really deep. You can either end up like running Black Friday sales, or running like Netflix with monkey enjoying tea. Running on cloud is no different than running on-premise, just you have to start all over again, because now you have to re-consider network, security, monitoring, and practice.

anonbanker 12y ago

it's about a 5-7 year cycle; IT services are outsourced (this time, to the cloud), but as time progresses, all of those services are consolidated (usually, under new management) in-house. I used to be able to set a watch by HP's cycle before the EDS buyout.

forgottenacc56 12y ago

I'm telling you theres real opportunity for someone to create "the Uber of colocation/dedicated hosting".

superuser2 12y ago
OpenStack was supposed to facillitate this. Doesn't seem to get much use.
- AndyNemmity 12y ago
  It does from my perspective. I could list many companies that use it for major things. It doesn't always get publicity. I work for a major company, and we don't scream from the roof tops about it.

anthony_barker 12y ago

In my experience firewalls and network is more expensive than servers.... Anyone offer competition to VPC?

kovyrin 12y ago
Softlayer manages the network for us. They offer a fully isolated backend network (frontend connection to the world is optional). For frontend connections we simply use iptables.

shimon_e 12y ago

RunAbove from OVH seems to be the best of both worlds. You can rent dedicated hardware by the hour for very reasonable prices.

kovyrin 12y ago
AFAIK Softlayer has hourly-based rental for real hardware as well. Never tried that, so could not comment on it though.

johnnycarcin 12y ago

Interesting article, thanks for sharing. We actually just went the exact opposite because of the larger scale issues we were having with Softlayer. Do you feel like you lost any resiliency by making the switch to physical servers (more virtual instances on one physical server, servers in the same rack, etc)?

kovyrin 12y ago
No, I really do not think going to EC2 could be beneficial in any way in terms of improving resiliency compared to Softlayer. SL allows you to control which VLANs your box will end up on. VLANs could be treated as racks (since they do not allocate more than one VLAN per rack). Then you have multiple DCs in one region (e.g. DAL01, DAL05, DAL07, etc) and you have many different regions (DAL, SEA, WAS, AMS, etc). I'd be very interested what problems you were having with them and at what scale. If this is a private topic, we could do it over email or some other medium if you like. You can contact me by any of the means listed here: http://kovyrin.net/contact/ http://kovyrin.net/contact/
- johnnycarcin 12y ago
  We were about 75% virtual with SL and 25% bare metal. One of our issues with the virtual stuff is when we started dedicating them to a set VLAN, multiple times we ran into an issue where some type of resource for that pod the VLAN was in would be maxed out (usually storage) so we couldn't create a new instance. The solution we were given was to let the system pick a VLAN but by doing that we had lost control of the placement and added some complexity to our architecture. Aside from that it was mainly nit-picky type stuff, but still things that were annoying (networking issues between DCs, networking issues between pods, internal mirrored apt-get repos going out of sync, API is kind of blah, etc). We use docker so having a few bare metal machines with tons of containers on them wasn't a great HA setup (for us at least), even running in two data centers. The fairly quick setup time though was a nice selling point. When we went to AWS things just kind of worked. The API was easier to use and the GUI portal was way nicer/stable. So far we have not had any odd issues with our instances, but we also typically run them at about 50% capacity so that might be why. It is also still early so maybe things will come up in 6+ months that send us back to SL :)

peterwwillis 12y ago

Cases where people try to use EC2: - We're starting from scratch and think AWS will give us flexibility for cheap - We have existing servers and think moving from them to AWS will give us flexibility for cheap Cases where people move away from EC2: - It was slow/unreliable - It was expensive Conlusion: You should use AWS EC2 in order to save money and have flexible resource allocation, but don't expect it to be stable or cost-effective.

tschellenbach 12y ago

From my experience AWS is one of the most reliable hosting providers. It's extremely easy to setup a fault tolerant infrastructure using cloudformation, puppet/chef etc, boto and a bit of autoscaling. The only disadvantage AWS has is costs. In many cases it's more expensive than a traditional hosting provider. On the flip side, your engineers will be more productive. What I miss in this article is any details on why they had issues with AWS. You can't just say it wasn't reliable and not explain the details. AWS works for all of the world's largest startups, why didn't it work for Swiftype?

stephen-mw 12y ago

Here's my anecdotal, one-data-point experience from moving a giant EC2 environment to datacenter: 1. Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. 2. Any weirdness you experienced with AWS infrastructure will be replaced with weirdness in your own environment, except now you're on the line to troubleshoot and fix it yourself. 3. Operation staff will immediately start guarding the food bowl as resources become finite. Server provision waits start to seem like breadlines. Power is consolidated with Those Whom You Must Ask. 4. Your cost will decrease, sometimes significantly so. 5. You'll have more hardware flexibility to run your app just the way you want to (Stack Overflow's mega databases come to mind[0]). In the end I think this type of transition is for stable companies that don't mind or even prefer strong divisions of labor (coders who code, sysadmins who sysadmin, testers who test), but it's not for startups or companies that hope to move with any kind of strong velocity. [0] http://highscalability.com/blog/2014/7/21/stackoverflow-update-560m-pageviews-a-month-25-servers-and-i.html http://highscalability.com/blog/2014/7/21/stackoverflow-upda...

chx 12y ago
> Your operational overhead will increase _a lot_. Be ready to hire on a lot of ops staff if you expect them to do anything but put out fires. And as you grow you'll need experts, people like network engineers. Why? What are you talking about? You are hiring servers you are no colo'ing them. Networking them is not your problem. Your responsibility still starts from a root prompt just there's no VM layer between that and the physical server.
- kovyrin 12y ago
  Yes, it really does look like @stephen-mw is talking about a migration to an owned/colocated hardware. And that's a very different kind of hairy mess I'd prefer to stay away from as long as humanly possible.
  - chx 12y ago
    OMG, colo. I've been the architect of a Top 500 website acquired by a Top 100 and even for that we just rented a few servers (you'd be surprised how few -- a dozen or so). I do not really know how big you need to be make colo a sensible choice but quite probably much, much bigger than most think of. We ran the numbers and they weren't pretty. And for cloud: we purchased video conversion as a service, that one ran in the cloud. I can see how that makes sense.

deitcher 12y ago

Oleksiy, can you share the economics? You said 50% savings. What was everything you had running in EC2 (easy to figure out the costs), what were your equivalents in softlayer? Would be very interesting to see the economics.

icedchai 12y ago

I'm convinced hybrid cloud is the way to go. Anything needing high IO performance should be on dedicated. Anything needed CPU/memory elasticity (worker nodes, etc) should be in the cloud. Assuming you can get low latency connectivity into AWS with DirectConnect, this might work?

kovyrin 12y ago
Yeah, that (worker nodes, async processing, etc) seems like one of those ideal use-case for clouds that I could definitely support as a viable option for companies that aren't ready for all-in cloud deployment.
fleitz 12y ago
Cpu also generally sucks due to cache thrashing on non-trivial datasets
- icedchai 12y ago
  Heh. So what is a good use case for EC2?
  - fleitz 12y ago
    When you have highly variable load. eg. Netflix probably spins up thousands of servers for a few hours.
nasalgoat 12y ago
The speed of light is not your friend. You can never compete with the latency of a local network versus even the fastest connection to a cloud provider.

amazon_not 12y ago

The biggest problem with AWS is the outrageous cost of bandwidth. Even if you ignore all the other cost differentials, the bandwidth charges will kill you at scale. Unfortunately cloud computing, or at the very least AWS, overpromises and underdelivers at scale. All in all economically viable use cases for cloud computing are very few and very specific at scale.

wmf 12y ago
Since people are sharing their experiences in this thread, has anyone tried using Direct Connect to get cheaper bandwidth?
- amazon_not 12y ago
  There aren't a lot of savings to be had. In addition to paying for the Direct Connect infrastructure you still have to pay Amazon per GB charges, albeit at a slightly lesser rate. The only real solution is to move the bandwidth usage off AWS.

qaq 12y ago

Finally sane take on Cloud BS :)

hashtree 12y ago

A concrete example of how I saved a few hundred thousand dollars over the AWS by building quarter rack colocation setups with SuperMicro servers: https://gist.github.com/rockymadden/5561377 https://gist.github.com/rockymadden/5561377 With Ansible, I spend no more than an hour a week, amortized, maintaining both the hardware and administration. I assume nodes for any specific role will fail, I only scale horizontally, I always have redundancy for every role, I stay off disk as long as possible (heya 512GB RAM redis cluster), etc.

danieltillett 12y ago
Great resource. Are you going to keep this updated?

abalone 12y ago

"2-3 serious outages a week" with EC2?? Details, please.

kovyrin 12y ago
Those weren't outages for a specific instance. But from the whole pool of instances we were running 2-3 would have networking issues, random unexplained hangs requiring an instance restart, huge CPU performance drops, IO hangs, etc, etc.

billyhoffman 12y ago

I used Rackspace (and later EC2) originally because of upfront capital costs. I bootstrapped my company, and paying a thousand dollars or more, plus hosting/racking fees, just wasn't doable. $50-100 bucks for a reasonable Windows server? Easy deal to make when cash flow is small.

jonpress 12y ago

I think cloud providers are mostly good for small/medium, fast growing startups. Big companies need more granular flexibility. The path to vertical integration in the software indsutry is a lot smoother than in many other industries, there are many levels: You can... 1. Make your own hardware 2. Own/manage your own hardware 3. Rent commodity hardware from a standard hosting provider 4. Use IaaS (e.g. EC2) 5. Use PaaS (e.g. AppFog, Nodejitsu) 6. Use BaaS (e.g. Firebase, PubNub, Pusher.com) The higher the level, the more technical flexibility you lose. The bigger the company, the more it makes sense operate at a lower level because there is no significant wastage being introduced as you move down the levels (you remove the middlemen so you can pocket their profits) and the capital cost to move between levels is relatively low. If you compare software to another industry like cheesemaking for example, if you're a cheesemaker and you want to make your own milk, the next step is to buy the whole farm and then you have to figure out what do do with the meat (wastage). Going between these two levels is expensive and could mean doubling or tripling your expenses so it's not an easy move to make.

deleted 12y ago

[deleted]

jamiesonbecker 12y ago

Cost-wise, AWS makes a lot of sense when growth is not easy to project, happens rapidly, or varies (seasonally, randomly, etc). Of course, some of this has to do with your team's skill level, but I've had clients run up $100k+ monthly bills at AWS with a relatively small build-out. (and, wow, VPC migrations..) For fixed or predictable growth patterns on a mature app/platform, a slow build-out on real iron will generally be significantly less expensive, all other things being equal. However, there are other advantages to AWS that gets lost in this story, such as pre-built, highly scalable datastores. Comparing EC2 to real iron misses most of the real story on why the cloud is changing everything. One of the hardest things I have to tell clients is not to build their own datastore/database in house or at EC2; sometimes the case is clearcut, and sometimes not so much, but if you have a datastore at AWS that gives you 80% of what you need, use it instead of rolling your own. (source: IAMA AWS Growth Architect)

BackOel 12y ago

Thanks for the information