4 ms·

There is a lot of votes here for cloud providers. And while I think AWS is great, it is not great at everything. What I've done recently is buying used servers

by olavgg 10y ago

There is a lot of votes here for cloud providers. And while I think AWS is great, it is not great at everything. What I've done recently is buying used servers and 10G/40g network switches from Ebay and rented a colo(colocation) rack, which can be had from $500-$1000 per month per rack. This often includes 100mbit++ internet, power, cooling and more (https://en.wikipedia.org/wiki/Colocation_centre https://en.wikipedia.org/wiki/Colocation_centre) This has been the most cost effective way for me to deploy for example Hadoop, Ceph, Elasticsearch, Huge Varnish cache solutions. While I understand this is not for everyone, it is absolutely something to consider if you have a strong devops team. I do all this myself as I've build my own automation tools over the years that simplify setup and monitoring. On the other hand, I use AWS for GPU instances as I find it very cost effective because it is very easy to scale up and down by demand. And investing in this kind of hardware is expensive/risky. The power efficiency / performance is still following Moores law for each year for GPU's, and I expect new hardware that is better optimized for neural networks / machine learning is just around the corner.

osipov 10y ago
I did a cost analysis of Amazon GPUs vs. Softlayer GPUs and the latter gave me a better price/performance ratio.
- imroot 10y ago
  But what you'll get in performance, you'll lose in sanity. I know that everyone has their "Horror" stories about a provider that they'd avoid at all costs, but, Softlayer still has an IBM "Mindset." Provisioning servers? You ask them, then their sales team approves -- hardly automatic. Seemingly minor changes to their system will force a hypervisor restart, and dealing with their account reps for anything is a nightmare. I'm not saying that Amazon is the end-all, be-all; I'm just saying that the grass isn't greener on the softlayer side of the fence.
  - sokoloff 10y ago
    Softlayer made my "do not do business with again" list after about 3 years of renting a server from them. I cancelled the server within what I believed (and still believe) to be the time required and they billed me an extra month, refusing to acknowledge the cancellation was timely submitted. They sit right behind Oracle and Verisign on the "never again" list.
applejuicestand 10y ago
How does the cost of doing it yourself compare to AWS for what you need?
tokenizerrr 10y ago
How often do you have hardware failures? How do you deal with that? You mention hadoop, so just rotate slaves?
jsingleton 10y ago
Colo is certainly a cheap way to do things for a steady load, and you get great performance for the money, but who wants to get paged at 4am? I know HN and StackOverflow do things this way but I don't even want to manage the OS these days. We used to host our servers on-site and it was a massive pain. At least with colo you don't need to personally worry about power or internet going down, but you still need to replace dead disks.
- CaptSpify 10y ago
  Curious as to why you think aws means you won't get paged at 4-am? If you have a redundant setup, you won't get paged as everything can just fail-over.
  - jsingleton 10y ago
    If you set things up badly then you can certainly get paged when on AWS. If you set things up correctly (multiple AZs / regions and auto scaling etc.) then it's much less likely. There could of course be software issues but hardware is no longer your problem and you can let experts do it for you. Yes, you could do this yourself but that would cost more and reduce the advantage. How many colo racks in how many data centres / countries do want to have to visit? Even with redundant hot spare fail-over you will need to fix the hardware at some point and will be running at risk until you do. A hybrid approach can work, with colo for the base load and cloud for scaling spikes / redundancy.
    - CaptSpify 10y ago
      > hardware is no longer your problem and you can let experts do it for you. I've heard this before, but it's basically not true. You still have hardware problems, you're just paying to pretend they aren't there anymore. If you want to do that, it's OK, but we need to stop pretending that throwing money at a problem makes it go away. > Yes, you could do this yourself but that would cost more and reduce the advantage. Compared to AWS pricing? No thanks. I've yet to personally see a company save money by using AWS (ymmv of course). > Even with redundant hot spare fail-over you will need to fix the hardware at some point and will be running at risk until you do. Sure, but that's generally where you save your money. And if you do it right, it doesn't have to be at 4 am
boulos 10y ago
If what you're after is 24x7 always on machines in a single DC, you're absolutely right: both GCE and EC2 are charging you for a premium you're not using (scale up and down, refresh your machines, handle power failures and maintenance, geographic diversity). That said, I think your first two examples (Hadoop and Ceph) are competively priced on Google Cloud. With Dataproc, you can put all your HDFS data on GCS and then run your cluster on Preemptible VMs. Combined with the ability to spin up and down as you need it, that's actually hard to beat economically in a colo setup. Disclosure: I work on Google Cloud.
- Veratyr 10y ago
  > I think your first two examples (Hadoop and Ceph) are competively priced on Google Cloud. I think this depends on: - Whether Ceph is being used as block storage (RBD) or object storage (regular Rados, RGW) - How much bandwidth is being egressed I don't know much about block storage on the GCE but for object storage, it's not hard to beat GCS pricing by buying a few second hand SC846 and filling them with WD Red drives. The biggest issue with Google's cloud services (and most others actually) for me though is that network egress is _horrendously_ expensive compared to the kind of deal you can get buying transit directly. Google's network is also one of the best of the planet but not everyone wants or needs to pay for that. At the highest volume pricing, GCP egress costs $0.08/GB. Even with CDN interconnect pricing and free egress through CloudFlare, you're paying $0.04/GB. Contrast this to a budget colo like Joe's Datacenter (joesdatacenter.com) where you pay $75/mo for 100Mbps 95th percentile. That's $75/mo for 32.4TB of egress ($0.002/GB), which would cost $1280 on GCP. At the higher volumes, you can get 10Gbps from FDC Servers for $1k/mo. Saturating that port would get you 3.24PB of traffic in a month, costing $0.0003/GB (a hundredth of what you'd pay Google). This means if you want to, for example, store your data on GCS and do your compute elsewhere (or vice versa), you're paying far more for premium bandwidth than you really need to. It's a really annoying kind of lockin.

by olavgg 10y ago

osipov 10y ago

I did a cost analysis of Amazon GPUs vs. Softlayer GPUs and the latter gave me a better price/performance ratio.

imroot 10y ago
But what you'll get in performance, you'll lose in sanity. I know that everyone has their "Horror" stories about a provider that they'd avoid at all costs, but, Softlayer still has an IBM "Mindset." Provisioning servers? You ask them, then their sales team approves -- hardly automatic. Seemingly minor changes to their system will force a hypervisor restart, and dealing with their account reps for anything is a nightmare. I'm not saying that Amazon is the end-all, be-all; I'm just saying that the grass isn't greener on the softlayer side of the fence.
- sokoloff 10y ago
  Softlayer made my "do not do business with again" list after about 3 years of renting a server from them. I cancelled the server within what I believed (and still believe) to be the time required and they billed me an extra month, refusing to acknowledge the cancellation was timely submitted. They sit right behind Oracle and Verisign on the "never again" list.

applejuicestand 10y ago

How does the cost of doing it yourself compare to AWS for what you need?

tokenizerrr 10y ago

How often do you have hardware failures? How do you deal with that? You mention hadoop, so just rotate slaves?

jsingleton 10y ago

Colo is certainly a cheap way to do things for a steady load, and you get great performance for the money, but who wants to get paged at 4am? I know HN and StackOverflow do things this way but I don't even want to manage the OS these days. We used to host our servers on-site and it was a massive pain. At least with colo you don't need to personally worry about power or internet going down, but you still need to replace dead disks.

CaptSpify 10y ago
Curious as to why you think aws means you won't get paged at 4-am? If you have a redundant setup, you won't get paged as everything can just fail-over.
- jsingleton 10y ago
  If you set things up badly then you can certainly get paged when on AWS. If you set things up correctly (multiple AZs / regions and auto scaling etc.) then it's much less likely. There could of course be software issues but hardware is no longer your problem and you can let experts do it for you. Yes, you could do this yourself but that would cost more and reduce the advantage. How many colo racks in how many data centres / countries do want to have to visit? Even with redundant hot spare fail-over you will need to fix the hardware at some point and will be running at risk until you do. A hybrid approach can work, with colo for the base load and cloud for scaling spikes / redundancy.
  - CaptSpify 10y ago
    > hardware is no longer your problem and you can let experts do it for you. I've heard this before, but it's basically not true. You still have hardware problems, you're just paying to pretend they aren't there anymore. If you want to do that, it's OK, but we need to stop pretending that throwing money at a problem makes it go away. > Yes, you could do this yourself but that would cost more and reduce the advantage. Compared to AWS pricing? No thanks. I've yet to personally see a company save money by using AWS (ymmv of course). > Even with redundant hot spare fail-over you will need to fix the hardware at some point and will be running at risk until you do. Sure, but that's generally where you save your money. And if you do it right, it doesn't have to be at 4 am

boulos 10y ago

If what you're after is 24x7 always on machines in a single DC, you're absolutely right: both GCE and EC2 are charging you for a premium you're not using (scale up and down, refresh your machines, handle power failures and maintenance, geographic diversity). That said, I think your first two examples (Hadoop and Ceph) are competively priced on Google Cloud. With Dataproc, you can put all your HDFS data on GCS and then run your cluster on Preemptible VMs. Combined with the ability to spin up and down as you need it, that's actually hard to beat economically in a colo setup. Disclosure: I work on Google Cloud.

Veratyr 10y ago
> I think your first two examples (Hadoop and Ceph) are competively priced on Google Cloud. I think this depends on: - Whether Ceph is being used as block storage (RBD) or object storage (regular Rados, RGW) - How much bandwidth is being egressed I don't know much about block storage on the GCE but for object storage, it's not hard to beat GCS pricing by buying a few second hand SC846 and filling them with WD Red drives. The biggest issue with Google's cloud services (and most others actually) for me though is that network egress is _horrendously_ expensive compared to the kind of deal you can get buying transit directly. Google's network is also one of the best of the planet but not everyone wants or needs to pay for that. At the highest volume pricing, GCP egress costs $0.08/GB. Even with CDN interconnect pricing and free egress through CloudFlare, you're paying $0.04/GB. Contrast this to a budget colo like Joe's Datacenter (joesdatacenter.com) where you pay $75/mo for 100Mbps 95th percentile. That's $75/mo for 32.4TB of egress ($0.002/GB), which would cost $1280 on GCP. At the higher volumes, you can get 10Gbps from FDC Servers for $1k/mo. Saturating that port would get you 3.24PB of traffic in a month, costing $0.0003/GB (a hundredth of what you'd pay Google). This means if you want to, for example, store your data on GCS and do your compute elsewhere (or vice versa), you're paying far more for premium bandwidth than you really need to. It's a really annoying kind of lockin.