3 ms·

I'm curious about what is behind Moz's statement "The business impact is profound. We’re spending less and have improved reliability and efficiency." I don't h

by apapli 13y ago

I'm curious about what is behind Moz's statement "The business impact is profound. We’re spending less and have improved reliability and efficiency." I don't have the experience with large AWS setups to comment about the cost implications, but improved reliability and efficiency strikes me as questionable - AWS's reliability has improved markedly over the last 18 months to the point I cannot remember when their last major outage was. I'm keen to hear other peoples' thoughts on this..

justinsb 13y ago
There may not have been a recent total outage of the kind that hits the news, but I would bet Moz are referring to degraded service (slow CPU/disk/network) that causes their application to fail (or their error rate to rise). I'm not very familiar with Moz, but it looks like they have an analytics product, which is probably particularly sensitive to slowdowns.
- res0nat0r 13y ago
  Sounds like they are more at a size that makes sense to have an on site setup vs. a cloud based one. The cloud market is maturing now and a hybrid private/public model I think is going to become more of the norm, for companies that are large enough that this makes sense. It is cheaper to run your own hardware when you are big enough, and then use AWS for failover / load spikes / cool tech, etc.
  - corresation 13y ago
    The scale where doing your own thing makes financial sense can be much lower than many think. About two thousand dollars a month gets half a rack with a high-speed, highly-reliable connection in co-location. The lease payments on several extremely high performance, flash storage rack servers would be a couple thousand dollars a month. Throw on some shared nothing redundant virtualization, and you have the equivalent of an army of 4xmegaxlargemongos. AWS has a place (such as being the redundant offsite backup for that scenario, or for extremely variable service requirements), but endlessly on here we see people disbelieving alternatives make sense.
    - AznHisoka 13y ago
      I own 15 OVH servers (32 GB RAM, 2 TB HD, quad-core), and the cost monthly is less than $1000. What can that buy more in Amazon? Maybe a huge instance? But I surely can't run thousands of distributed crawlers and store a distributed search index.
      - gtaylor 13y ago
        You can get some decent beef with the new C3 instances, but you have to pay a good chunk of the year upfront (called a reservation) to get the sane rates. If you do that, the prices are OK. But you don't come to AWS just for EC2 instances. You're there to combine their portfolio of services in various ways. Because of this, it'd probably be wrong to just compare $1000 on AWS vs what you are paying now.
  - deleted 13y ago
    [deleted]
- iamchrisle 13y ago
  Moz is an analytics startup for marketers. It essentially crawls the internet to create a graph of links and reports that in their user interface. They also calculate a score based on the authority of links similar but not the same as Google's Pagerank. They also do social tracking. Technical details: I don't know exactly how they do it but my guess is that they are I/O bound on the crawling side, then CPU bound on the parsing and processing side. I'm assuming they use different machines for those tasks. On the crawling side, the index with 60-70 billion URLS was using 80 cc2.8xlarge machines with a backup on 200 c1.xlarge machines. (http://moz.com/blog/one-step-back-two-steps-forward http://moz.com/blog/one-step-back-two-steps-forward) (DISCLAIMER: I'm in the same industry. I am a developer with working for an indirect competitor. I know a few engineers and non-engineers who work at Moz.)
  - AznHisoka 13y ago
    Rand replied that they don't use AWS for crawling. I would imagine the main bottleneck, by far is the I/O with reading/writing to the database clusters and search index. Crawling is relatively cheap, it's storing that data that's hard.
adrianpike 13y ago
A big thing you fight with services like AWS is a lack of transparency - everything's very opaque, and so when you have something like an IO degradation on a given node, you've got very little to go off of, and you have to try and decipher if it's that you're seeing neighbor pressure on that node, if you're on degraded hardware, or if you're just abusing the hardware inappropriately. Moving to your own hardware lets you actually reason about what's going on in a controlled environment, and at the scale that Moz is working at allows us to get hardware that's robust, well-supported, and reliable enough to have a pretty serious net positive overall.

by apapli 13y ago

justinsb 13y ago

There may not have been a recent total outage of the kind that hits the news, but I would bet Moz are referring to degraded service (slow CPU/disk/network) that causes their application to fail (or their error rate to rise). I'm not very familiar with Moz, but it looks like they have an analytics product, which is probably particularly sensitive to slowdowns.

res0nat0r 13y ago
Sounds like they are more at a size that makes sense to have an on site setup vs. a cloud based one. The cloud market is maturing now and a hybrid private/public model I think is going to become more of the norm, for companies that are large enough that this makes sense. It is cheaper to run your own hardware when you are big enough, and then use AWS for failover / load spikes / cool tech, etc.
- corresation 13y ago
  The scale where doing your own thing makes financial sense can be much lower than many think. About two thousand dollars a month gets half a rack with a high-speed, highly-reliable connection in co-location. The lease payments on several extremely high performance, flash storage rack servers would be a couple thousand dollars a month. Throw on some shared nothing redundant virtualization, and you have the equivalent of an army of 4xmegaxlargemongos. AWS has a place (such as being the redundant offsite backup for that scenario, or for extremely variable service requirements), but endlessly on here we see people disbelieving alternatives make sense.
  - AznHisoka 13y ago
    I own 15 OVH servers (32 GB RAM, 2 TB HD, quad-core), and the cost monthly is less than $1000. What can that buy more in Amazon? Maybe a huge instance? But I surely can't run thousands of distributed crawlers and store a distributed search index.
    - gtaylor 13y ago
      You can get some decent beef with the new C3 instances, but you have to pay a good chunk of the year upfront (called a reservation) to get the sane rates. If you do that, the prices are OK. But you don't come to AWS just for EC2 instances. You're there to combine their portfolio of services in various ways. Because of this, it'd probably be wrong to just compare $1000 on AWS vs what you are paying now.
- deleted 13y ago
  [deleted]
iamchrisle 13y ago
Moz is an analytics startup for marketers. It essentially crawls the internet to create a graph of links and reports that in their user interface. They also calculate a score based on the authority of links similar but not the same as Google's Pagerank. They also do social tracking. Technical details: I don't know exactly how they do it but my guess is that they are I/O bound on the crawling side, then CPU bound on the parsing and processing side. I'm assuming they use different machines for those tasks. On the crawling side, the index with 60-70 billion URLS was using 80 cc2.8xlarge machines with a backup on 200 c1.xlarge machines. (http://moz.com/blog/one-step-back-two-steps-forward http://moz.com/blog/one-step-back-two-steps-forward) (DISCLAIMER: I'm in the same industry. I am a developer with working for an indirect competitor. I know a few engineers and non-engineers who work at Moz.)
- AznHisoka 13y ago
  Rand replied that they don't use AWS for crawling. I would imagine the main bottleneck, by far is the I/O with reading/writing to the database clusters and search index. Crawling is relatively cheap, it's storing that data that's hard.

adrianpike 13y ago

A big thing you fight with services like AWS is a lack of transparency - everything's very opaque, and so when you have something like an IO degradation on a given node, you've got very little to go off of, and you have to try and decipher if it's that you're seeing neighbor pressure on that node, if you're on degraded hardware, or if you're just abusing the hardware inappropriately. Moving to your own hardware lets you actually reason about what's going on in a controlled environment, and at the scale that Moz is working at allows us to get hardware that's robust, well-supported, and reliable enough to have a pretty serious net positive overall.