6 ms·

I have posted a few times about how absurdly expensive all the cloud providers are. If you have a baseline load you should be co-locating bare metal. Any exces

I have posted a few times about how absurdly expensive all the cloud providers are. If you have a baseline load you should be co-locating bare metal. Any excess capacity you need should spill over into your AWS/GCE/Azure account. For example: A dedicated m4.16xLarge EC2 instance in AWS is $3987/month. You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx), and colo it with a 1GB/s blended bandwidth connection billed at the 95th percentile for $150/month.

api 9y ago
OVH has absurdly low prices on both. If you have a baseline load take a look. Their enterprise offerings are pretty rock solid and they have multiple data centers with direct fiber backplane interconnects. IMHO the big cloud providers only make sense if you have a highly elastic load or if you are making extensive use of their as-a-service higher level offerings. If you're running baseline load bare metal destroys them.
- TheDrizzle43 9y ago
  Unfortunately OVH doesn't have any data centers in the US although they seem to be working on them.
  - api 9y ago
    Montreal has low pings to the US.
- eropple 9y ago
  I used to use OVH on occasion, but I was really disappointed with their North American-based tech support. Didn't seem particularly plugged-in and English proficiency was shaky. (At this point I'm in AWS because I don't really care about low-point performance and would rather spike up when needed.)
mbesto 9y ago
> You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx), and colo it with a 1GB/s blended bandwidth connection billed at the 95th percentile for $150/month. And then magically maintain it for free?
- orik 9y ago
  hopefully with your spare 3437 dollars a month you could figure out how to ever maintain such a thing.
  - Klathmon 9y ago
    Where can I find competent sysadmins capable of managing a system like that on their own for $40,000 a year?
    - baudehlo 9y ago
      You can buy sysadmin services. This isn't a full time job at that level.
      - Klathmon 9y ago
        Then you are just trading one fee for another, but the "sysadmin services" you are ordering most likely has slower response time, less resources, less knowledge, and worse uptime guarantees than someone like aws. Yeah, at the larger sizes it might make sense, but not for everyone.
        dismantlethesun 9y ago
        At smaller sizes too. If you just have one server, it's unlikely that the slower response time will loose you 3000 dollars per month which is the savings of dedicated service versus aws. Many small businesses run on leased hardware with on site support. Rackspace for instance made their fortune in that sector.
        9y ago
gfody 9y ago
you can also pickup an enterprise flash device and have a few terabytes of 500k+IOPs for a few grand whereas reaching a similar performance profile via provisioned IOPs will cost you 10x as much every month.
- Johnny555 9y ago
  You're comparing apples to oranges -- if you want a couple TB of locally attached storage, you'd get an i3.2xlarge for $450/month with 1.9TB of NVMe SSD (plus 8 cores CPU + 60GB RAM). Using EBS volumes with provisioned IOPS give you some advantages that you don't get with a bare SSD -- like snapshot support and the ability to easily detach it and move it to another server.
  - gfody 9y ago
    you're right, the performance of the local storage on i3 instances is a lot better than normal EBS but still nothing close to what you would get from just buying an Intel P4500 or similar though. http://www.aerospike.com/benchmarks/amazon-ec2-i3-performance-results/ http://www.aerospike.com/benchmarks/amazon-ec2-i3-performanc...
    - user5994461 9y ago
      Double the P4500. The raid is not gonna store data out of thin air.
      - user5994461 9y ago
        And add a third one while you're at it. Given the price of the server and the data you have on it, it's important enough to warrant a hot spare.
Johnny555 9y ago
I'm not sure which region you priced, but an m4.16xlarge in us-east-1 (Virginia) is $3.20/hour, or $2304/month. But if you are comparing it to a purchased server, you ought to be looking at Reserve pricing, a 1 year reserved instance is $16K/year, 3 years is $32K (or you can pay monthly if you want). Still not as cheap as owning the server, but not quite as bad.
- shiftpgdn 9y ago
  I had selected a dedicated instance. It's been my experience the difference in performance between dedicated and non dedicated is dramatic. edit: I had selected a dedicated instance in order to compare apples to apples. By definition you don't share a colo server.
  - dhillonsandeep 9y ago
    this is not true, there should be no difference
  - openasocket 9y ago
    Why get a dedicated instance and not get reserve pricing? You can get those instances for a fraction of what you're currently paying
  - Johnny555 9y ago
    With an m4.16xl, there's no effective difference between a dedicated or default tenancy instance - only one .16xl can run on a physical hypervisor, which is reflected in the pricing, it's $3.20/hour whether it's dedicated or not: https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/ https://aws.amazon.com/ec2/purchasing-options/dedicated-inst... For smaller instances, there is around a 10% increase for dedicated, for example, for an m4.4xl, the price goes to $0.88 for dedicated tenancy versus $0.80 for default. There is a $2/hour fee for each region in which you have dedicated instances, but that gets lost in the noise if you're using significant AWS resources. (it's $2/hour regardless of how many instances you're running) We haven't noticed any significant difference between performance on dedicated and non-dedicated instances, and only use dedicated instances where required for compliance/regulatory reasons.
philipov 9y ago
[Off Topic] I have a chrome extension that replaces all occurrences of "the cloud" with "my butt". It is the best extension.
- eip 9y ago
  Does it also replace 'millennials' with 'snake people'?
didip 9y ago
1. Will Dell servers magically appear in your racks all mounted correctly? Shipping delays is a very serious and real problem. 2. Will Dell servers arrived with 0 problems on its components? Any time I ordered meaningful amount of servers, I usually get about 5% fail rate. VMs aren't perfect but you can destroy and recreate in different region almost real time. 3. Ever had to deal with difficult-to-work-with network admin? The cloud is significantly less pain in the ass.
- MBCook 9y ago
  #3 is why we went to Amazon, and its one of the reasons we were successful. Sad as that was. No weeks/months of delays/hassles/fights/sabotage. Just success as fast as we could do our part and figure out the few small hurdles.
  - _jal 9y ago
    ...That sounds more like an HR problem than a technical problem. There are network engineers in the world who don't live in caves and gnaw at the bones of administrative assistants.
    - MBCook 9y ago
      Oh it was, but it was the situation we had to operate in. And Amazon allowed to to achieve amazing things that never would have been possible and that's without the idea of things like Kinesis or Redshift that we couldn't have achieved easily/at all even without personnel problems. You simply could never get 3 new servers or drastically respec an existing one in under a day let alone minutes. We were a small shop so it's not like we had spare servers ready to go.
    - Spooky23 9y ago
      There are, but they are hard to find. Networking is a great place for assholes to build empires and exert control. In 20 years of professional engagement with distributed and data center networks, most of these guys (and they are almost always guys) running network orgs are an impediment and spend more time helping out their vendor of choice than anything else. I've run into awesome network guys in position of power only a few times, and they are 100x employees. The last major rollout of a service that I did was literally 3 months ahead of schedule solely because of the efforts of this awesome network dude snowman as recently put in charge.
InclinedPlane 9y ago
It's not the hardware you're paying for, it's the services. How much does it cost to employ a systems administrator? A lab tech? And so on. You see exactly the same pattern in software services. People pay more to buy products that are "turn key" versus having to hire someone to do tons of work building out systems or services or what-have-you. Also, hiring people is a risk, and it doesn't happen instantly. When you have a gap of months without a sysadmin things are bad. Hiring a bad sysadmin is even worse. Not to mention the impossibility of hiring 1/2 or 3/4 of a sysadmin. There's scaling limits there. If you have the resources to DIY, that can be a key competitive advantage, but not everyone does.
- icedchai 9y ago
  Often, you still need a "systems administrator" with the cloud. Except, now he calls himself a DevOps guy and charges a lot more. ;) AWS isn't going to configure itself.
user5994461 9y ago
Use google cloud or softlayer and the same machine is $1500 a month. It's no secret that AWS is a rip off on instance pricing. Use the competitors if you worry so much about pricing.
- vacri 9y ago
  How does that work out? Looking at the per-hour figures in AWS and GCE, they're similar for a 64-core, quarter terabyte RAM machine ($3.20 AWS, $3.40 GCE). Both vendors have ways to get your bill discounted (with google's being easier). AWS's instance pricing has been fairly comparable to other vendors for a couple of years now; it's their bandwidth that's costly. What are most people running anyway? Using this kind of behemoth server as a 'typical' example seems out of line.
  - user5994461 9y ago
    Google n1-standard-64 => $1710 AWS m4.16xlarge => $2592 Google base is slightly lower and it gives you 30% off automatically on instances that run 24/7 for the month. That's typical of on premise deployments. You get big servers because they are so annoying to deploy you might as well fill everything possible in the box. On the clouds, you would create a group of VM per service, with appropriate size for the service. It is significantly more efficient.
    - vacri 9y ago
      Isn't that $1710 already including the 30% discount? (It works out to $3.4 x 30ish days x 70%). You get similar discounts on AWS, but you have to commit for a year - given the OP's intent of leasing a Dell server for three years, this shouldn't be dismissed.
- deleted 9y ago
  [deleted]
vacri 9y ago
> A dedicated m4.16xLarge EC2 instance in AWS is $3987/month. You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx) Cherry-picking a bit there. You're entering a 3-year commitment with the Dell lease, but not with the EC2 lease. Take a 3-year RI on that EC2 instance, and the price drops considerably. If you also take away the 'dedicated' requirement, the 3-year RI is about twice as expensive, not ten times. Edit: correction from Johnny555 above: this top-tier server doesn't have a 'dedicated' option, so there's not an 'extra' cost for that. It's just twice the price, if you do a 3-year commitment (as you are doing for the dell)
jrochkind1 9y ago
You're paying for the staff time to set up the lower-level infrastructure and keep it running at _least_ as reliably as if you had your own staff on bare metal, quite likely more so. (including scaling it when you need more etc). It is interesting that they end up charging you this for _bandwidth_ though. Their staff time doesn't actually probably scale with bandwidth, but their prices do. I think it's probably kind of a way of pricing based on a proxy for customer size, likely revenue, and ability/willingness to pay. Rather than as a % margin on actual costs.
ceocoder 9y ago
When we account for cost of, 1) remote hands 2) network switches 3) hardware components you'll need for redundancy, I've personally replaced network cards because fans burned out or HDDs because of usual failure rate. 4) when I'm done using a n1-highcpu-32 I stop using it and not pay for it, or when I need 5 of those for 2 hours I provision them and stop when I'm done running a workload. 5) upgrades to hardware or additional hardware, intel released skylake, should I buy those and try to sell my Haswell on eBay? I'm not sure it becomes a simple price of server + colo cost comparison. With GCP (I use daily), AWS (I used to use) or Azure, what I'm paying for is someone else to focus on things like colo leases, capacity expansion, redundancy, physical security and provide higher order services - load balancers, BigTable/BigQuery, CloudSQL/RDS, GKE, DataProc. This allows me to run all of infra as close to capacity as possible while avoiding any wasted cycles However I do miss going to data centers and hear my machines hum and blink, that is quite the feeling.
Spooky23 9y ago
I just landed a few racks of bladecenters and storage. High 7 figure deal. This was probably the best implementation project ever... everything ran on time, the vendor delivered everything on time, facilities was 100% ready and pre-cabled, etc. I'm incredibly proud of our team -- our biggest issue was someone ordered the wrong SFP. Butter. Timeline: 120 days procurement (including product evaluation and PoC), 30 days install activity, 10 days buildout. 10 days software build, 15 days testing. That's a little over 6 months. A ton of vendor/build engineering/tester time. And we'll be at 50% capacity for another 4 months. Or... we could spin the whole thing up on AWS/GCE/Azure in < 30 days and only consume what we need when we need it.
- late2part 9y ago
  Which hardware vendor did you go with?

api 9y ago

OVH has absurdly low prices on both. If you have a baseline load take a look. Their enterprise offerings are pretty rock solid and they have multiple data centers with direct fiber backplane interconnects. IMHO the big cloud providers only make sense if you have a highly elastic load or if you are making extensive use of their as-a-service higher level offerings. If you're running baseline load bare metal destroys them.

TheDrizzle43 9y ago
Unfortunately OVH doesn't have any data centers in the US although they seem to be working on them.
- api 9y ago
  Montreal has low pings to the US.
eropple 9y ago
I used to use OVH on occasion, but I was really disappointed with their North American-based tech support. Didn't seem particularly plugged-in and English proficiency was shaky. (At this point I'm in AWS because I don't really care about low-point performance and would rather spike up when needed.)

mbesto 9y ago

> You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx), and colo it with a 1GB/s blended bandwidth connection billed at the 95th percentile for $150/month. And then magically maintain it for free?

orik 9y ago
hopefully with your spare 3437 dollars a month you could figure out how to ever maintain such a thing.
- Klathmon 9y ago
  Where can I find competent sysadmins capable of managing a system like that on their own for $40,000 a year?
  - baudehlo 9y ago
    You can buy sysadmin services. This isn't a full time job at that level.
    - Klathmon 9y ago
      Then you are just trading one fee for another, but the "sysadmin services" you are ordering most likely has slower response time, less resources, less knowledge, and worse uptime guarantees than someone like aws. Yeah, at the larger sizes it might make sense, but not for everyone.
      - dismantlethesun 9y ago
        At smaller sizes too. If you just have one server, it's unlikely that the slower response time will loose you 3000 dollars per month which is the savings of dedicated service versus aws. Many small businesses run on leased hardware with on site support. Rackspace for instance made their fortune in that sector.
      - 9y ago

gfody 9y ago

you can also pickup an enterprise flash device and have a few terabytes of 500k+IOPs for a few grand whereas reaching a similar performance profile via provisioned IOPs will cost you 10x as much every month.

Johnny555 9y ago
You're comparing apples to oranges -- if you want a couple TB of locally attached storage, you'd get an i3.2xlarge for $450/month with 1.9TB of NVMe SSD (plus 8 cores CPU + 60GB RAM). Using EBS volumes with provisioned IOPS give you some advantages that you don't get with a bare SSD -- like snapshot support and the ability to easily detach it and move it to another server.
- gfody 9y ago
  you're right, the performance of the local storage on i3 instances is a lot better than normal EBS but still nothing close to what you would get from just buying an Intel P4500 or similar though. http://www.aerospike.com/benchmarks/amazon-ec2-i3-performance-results/ http://www.aerospike.com/benchmarks/amazon-ec2-i3-performanc...
  - user5994461 9y ago
    Double the P4500. The raid is not gonna store data out of thin air.
    - user5994461 9y ago
      And add a third one while you're at it. Given the price of the server and the data you have on it, it's important enough to warrant a hot spare.

Johnny555 9y ago

I'm not sure which region you priced, but an m4.16xlarge in us-east-1 (Virginia) is $3.20/hour, or $2304/month. But if you are comparing it to a purchased server, you ought to be looking at Reserve pricing, a 1 year reserved instance is $16K/year, 3 years is $32K (or you can pay monthly if you want). Still not as cheap as owning the server, but not quite as bad.

shiftpgdn 9y ago
I had selected a dedicated instance. It's been my experience the difference in performance between dedicated and non dedicated is dramatic. edit: I had selected a dedicated instance in order to compare apples to apples. By definition you don't share a colo server.
- dhillonsandeep 9y ago
  this is not true, there should be no difference
- openasocket 9y ago
  Why get a dedicated instance and not get reserve pricing? You can get those instances for a fraction of what you're currently paying
- Johnny555 9y ago
  With an m4.16xl, there's no effective difference between a dedicated or default tenancy instance - only one .16xl can run on a physical hypervisor, which is reflected in the pricing, it's $3.20/hour whether it's dedicated or not: https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/ https://aws.amazon.com/ec2/purchasing-options/dedicated-inst... For smaller instances, there is around a 10% increase for dedicated, for example, for an m4.4xl, the price goes to $0.88 for dedicated tenancy versus $0.80 for default. There is a $2/hour fee for each region in which you have dedicated instances, but that gets lost in the noise if you're using significant AWS resources. (it's $2/hour regardless of how many instances you're running) We haven't noticed any significant difference between performance on dedicated and non-dedicated instances, and only use dedicated instances where required for compliance/regulatory reasons.

philipov 9y ago

[Off Topic] I have a chrome extension that replaces all occurrences of "the cloud" with "my butt". It is the best extension.

eip 9y ago
Does it also replace 'millennials' with 'snake people'?

didip 9y ago

1. Will Dell servers magically appear in your racks all mounted correctly? Shipping delays is a very serious and real problem. 2. Will Dell servers arrived with 0 problems on its components? Any time I ordered meaningful amount of servers, I usually get about 5% fail rate. VMs aren't perfect but you can destroy and recreate in different region almost real time. 3. Ever had to deal with difficult-to-work-with network admin? The cloud is significantly less pain in the ass.

MBCook 9y ago
#3 is why we went to Amazon, and its one of the reasons we were successful. Sad as that was. No weeks/months of delays/hassles/fights/sabotage. Just success as fast as we could do our part and figure out the few small hurdles.
- _jal 9y ago
  ...That sounds more like an HR problem than a technical problem. There are network engineers in the world who don't live in caves and gnaw at the bones of administrative assistants.
  - MBCook 9y ago
    Oh it was, but it was the situation we had to operate in. And Amazon allowed to to achieve amazing things that never would have been possible and that's without the idea of things like Kinesis or Redshift that we couldn't have achieved easily/at all even without personnel problems. You simply could never get 3 new servers or drastically respec an existing one in under a day let alone minutes. We were a small shop so it's not like we had spare servers ready to go.
  - Spooky23 9y ago
    There are, but they are hard to find. Networking is a great place for assholes to build empires and exert control. In 20 years of professional engagement with distributed and data center networks, most of these guys (and they are almost always guys) running network orgs are an impediment and spend more time helping out their vendor of choice than anything else. I've run into awesome network guys in position of power only a few times, and they are 100x employees. The last major rollout of a service that I did was literally 3 months ahead of schedule solely because of the efforts of this awesome network dude snowman as recently put in charge.

InclinedPlane 9y ago

It's not the hardware you're paying for, it's the services. How much does it cost to employ a systems administrator? A lab tech? And so on. You see exactly the same pattern in software services. People pay more to buy products that are "turn key" versus having to hire someone to do tons of work building out systems or services or what-have-you. Also, hiring people is a risk, and it doesn't happen instantly. When you have a gap of months without a sysadmin things are bad. Hiring a bad sysadmin is even worse. Not to mention the impossibility of hiring 1/2 or 3/4 of a sysadmin. There's scaling limits there. If you have the resources to DIY, that can be a key competitive advantage, but not everyone does.

icedchai 9y ago
Often, you still need a "systems administrator" with the cloud. Except, now he calls himself a DevOps guy and charges a lot more. ;) AWS isn't going to configure itself.

user5994461 9y ago

Use google cloud or softlayer and the same machine is $1500 a month. It's no secret that AWS is a rip off on instance pricing. Use the competitors if you worry so much about pricing.

vacri 9y ago
How does that work out? Looking at the per-hour figures in AWS and GCE, they're similar for a 64-core, quarter terabyte RAM machine ($3.20 AWS, $3.40 GCE). Both vendors have ways to get your bill discounted (with google's being easier). AWS's instance pricing has been fairly comparable to other vendors for a couple of years now; it's their bandwidth that's costly. What are most people running anyway? Using this kind of behemoth server as a 'typical' example seems out of line.
- user5994461 9y ago
  Google n1-standard-64 => $1710 AWS m4.16xlarge => $2592 Google base is slightly lower and it gives you 30% off automatically on instances that run 24/7 for the month. That's typical of on premise deployments. You get big servers because they are so annoying to deploy you might as well fill everything possible in the box. On the clouds, you would create a group of VM per service, with appropriate size for the service. It is significantly more efficient.
  - vacri 9y ago
    Isn't that $1710 already including the 30% discount? (It works out to $3.4 x 30ish days x 70%). You get similar discounts on AWS, but you have to commit for a year - given the OP's intent of leasing a Dell server for three years, this shouldn't be dismissed.
deleted 9y ago
[deleted]

vacri 9y ago

> A dedicated m4.16xLarge EC2 instance in AWS is $3987/month. You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx) Cherry-picking a bit there. You're entering a 3-year commitment with the Dell lease, but not with the EC2 lease. Take a 3-year RI on that EC2 instance, and the price drops considerably. If you also take away the 'dedicated' requirement, the 3-year RI is about twice as expensive, not ten times. Edit: correction from Johnny555 above: this top-tier server doesn't have a 'dedicated' option, so there's not an 'extra' cost for that. It's just twice the price, if you do a 3-year commitment (as you are doing for the dell)

jrochkind1 9y ago

You're paying for the staff time to set up the lower-level infrastructure and keep it running at _least_ as reliably as if you had your own staff on bare metal, quite likely more so. (including scaling it when you need more etc). It is interesting that they end up charging you this for _bandwidth_ though. Their staff time doesn't actually probably scale with bandwidth, but their prices do. I think it's probably kind of a way of pricing based on a proxy for customer size, likely revenue, and ability/willingness to pay. Rather than as a % margin on actual costs.

ceocoder 9y ago

When we account for cost of, 1) remote hands 2) network switches 3) hardware components you'll need for redundancy, I've personally replaced network cards because fans burned out or HDDs because of usual failure rate. 4) when I'm done using a n1-highcpu-32 I stop using it and not pay for it, or when I need 5 of those for 2 hours I provision them and stop when I'm done running a workload. 5) upgrades to hardware or additional hardware, intel released skylake, should I buy those and try to sell my Haswell on eBay? I'm not sure it becomes a simple price of server + colo cost comparison. With GCP (I use daily), AWS (I used to use) or Azure, what I'm paying for is someone else to focus on things like colo leases, capacity expansion, redundancy, physical security and provide higher order services - load balancers, BigTable/BigQuery, CloudSQL/RDS, GKE, DataProc. This allows me to run all of infra as close to capacity as possible while avoiding any wasted cycles However I do miss going to data centers and hear my machines hum and blink, that is quite the feeling.

Spooky23 9y ago

I just landed a few racks of bladecenters and storage. High 7 figure deal. This was probably the best implementation project ever... everything ran on time, the vendor delivered everything on time, facilities was 100% ready and pre-cabled, etc. I'm incredibly proud of our team -- our biggest issue was someone ordered the wrong SFP. Butter. Timeline: 120 days procurement (including product evaluation and PoC), 30 days install activity, 10 days buildout. 10 days software build, 15 days testing. That's a little over 6 months. A ton of vendor/build engineering/tester time. And we'll be at 50% capacity for another 4 months. Or... we could spin the whole thing up on AWS/GCE/Azure in < 30 days and only consume what we need when we need it.

late2part 9y ago
Which hardware vendor did you go with?