6 ms·

> Our TCO includes energy, hiring a part-time system administrator, and co-location costs. Which is a myopic view of TCO. This ignores so many things about pur

by mbesto 8y ago

> Our TCO includes energy, hiring a part-time system administrator, and co-location costs. Which is a myopic view of TCO. This ignores so many things about purchasing on-prem hardware (for good or bad): Admin: - Cost associated with finding an admin who understands how this thing works Speed / convenience: - Try before you buy - Time it takes for the box to be built, shipped and sent to the data center - Time (and cost) it takes to install software, drivers, etc Maintenance / Capitalization / Finances: - 3-year? What is the useful life of this? When does it seemingly become obsolete? - AWS will continually upgrade their hardware and you keep paying the same - Hardware can be capitalized, which means you can push it to the balance sheet (for tax or valuation purposes) - Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns. Hidden costs: - Returns, breakage, warranty in case of a hardware failure I understand why there is a market for this product, but it's not always an apples/orange comparison. Generally speaking, if you know what your workload is going to be, (I'd be hard pressed if a lot of orgs really know the answer to this) then on-prem hardware is not a terrible choice, but it has to be analyzed appropriately.

microcolonel 8y ago
> - Time (and cost) it takes to install software, drivers, etc I'd imagine this is part of having a system administrator.
vidarh 8y ago
> - Cost associated with finding an admin who understands how this thing works Managing AWS resources is not "free" either, or the market for people to handle devops work on AWS related to ongoing operations would be non-existent. My experience is when I've done consulting, clients on AWS used to end up paying more per instance than on-prem clients ended up paying per physical server. > - Time (and cost) it takes to install software, drivers, etc This is largely an initial cost of a day or two of setup when you start setting up an on-prem setup + for the first server of a totally new model. If you're buying one, then sure, you need to factor in a bit of time. If you're buying more, then if you use a competent admin the second one should be a matter of inserting boot media or (preferrably) configuring PXE booting, and picking an IP.
- fucking_tragedy 8y ago
  > My experience is when I've done consulting, clients on AWS used to end up paying more per instance than on-prem clients ended up paying per physical server. This has been my experience, as well. Clients who switch to AWS for cost-saving reasons will be disappointed.
  - mikemotherwell 8y ago
    ... unless the use case is weird, e.g. you need massive amounts of processing power infrequently.
    - vidarh 8y ago
      There certainly are people with use-cases like that, but it is rare to find people who both have large, sudden spikes that are not better handled in other ways (e.g. combination of CDNs and micro-caching), or where the load can not be trivially smoothed out enough (e.g. there's enough work with relaxed latency requirements or that can be scheduled to off peak times, or where the spikes are not small enough relative to base-load that it still pays off to put most load on dedicated equipment and use AWS or similar to handle spikes only. People often think they have workloads like that. I've done work on plenty of setups over the years were people were certain they had lots of big spikes, but most of them only had moderate day/night cycles (and usually a lot less spiky than they thought, because they'd never bothered to measure). Only in a few cases I've seen usage patterns that justified spinning instances up/down very often, and for those cases, ironically the availability of clouds have made dedicated hosting more cost effective, because if you host everything in VMs or containers anyway, you can now get to far higher utilization levels on your on-prem/colocated setups, and be prepared to spin up some cloud instances (automatically, if you wish, it's not that much effort), and tie them into your cluster when your load warrants it. It's not suitable for all workloads, but it works surprisingly well if anything. I do see that argument a lot for using cloud providers, but the vast majority of cloud setups I've seen end up having very static workloads growing at rates that gives plenty of time to prepare.
      - 8y ago
sabalaba 8y ago
(Author here) I agree with a lot of your points, especially the fact that TCO is practically impossible to calculate, is highly subjective, and should ideally include opportunity costs which, almost by definition vary from person to person. I've definitely thought through some of your points. First I want to say that if your load is spiky you have no business buying hardware. No matter how you slice it, you're simply not going to be able to get hundreds of GPUs of throughput with an 8 GPU machine so it's not possible to get the same "product". So for those doing short bursts of hyperparameter search, stick with cloud. But, like you said, for those with high utilization and known utilization patterns, it makes a lot of sense to go on-prem. Let's just say pretty much anybody who is buying a reserved instance from AWS should consider buying hardware instead. >- Cost associated with finding an admin who understands how this thing works Pricing based on what co location facilities will provide you without much search and is pretty generous. $10k per year per server is silly high and should cover that search cost. > - Try before you buy You can try essentially this exact machine on AWS. That instance and hardware are almost exactly the same. Now of course this doesn't apply to many instance types. > - Time it takes for the box to be built, shipped and sent to the data center 5 business days is our mean lead time for that unit. Most workstations are 2 days. > - Time (and cost) it takes to install software, drivers, etc It comes with that stuff pre-installed as mentioned in the article. See Lambda Stack for driver / framework / CUDA woes: https://lambdalabs.com/lambda-stack-deep-learning-software https://lambdalabs.com/lambda-stack-deep-learning-software > - 3-year? What is the useful life of this? When does it seemingly become obsolete? I'll admit it's a long time but 3 years is about right for GPUs from my experience. I personally purchased new GPUs for my workstations in 2012 (Fermi), 2015 (Maxwell), 2017 (Pascal). > - AWS will continually upgrade their hardware and you keep paying the same Yes, but only if you never commit to a reserved instance, in which case costs are 3x higher. If you buy a reserved instance you don't get upgraded hardware. > - Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns. Yea, most of these are people who are already using GPUs really often for internal training infra and are finding it expensive. > - Returns, breakage, warranty in case of a hardware failure Our hardware comes with a 3 year parts warranty. Of course, this doesn't pay for your time lost but it's pretty rare to see parts fail and the $10k / year more than covers in-colo swap outs. I agree with all of your points. I don't think that we really disagree with much here. TCO is hard to calculate :).
mippie_moe 8y ago
Hi, Lambda engineer here (I'm one of the authors). You bring up some good points, I'd like to address some of them: Admin: Time (and cost) it takes to install software, drivers, etc These tasks aren't made unnecessary by cloud. Yes, with cloud, once you've done the work, it can be forever encoded into an image or container. However, the same applies to on-prem using container solutions like Docker. Maintenance / Capitalization / Finances: 3-year? What is the useful life of this? When does it seemingly become obsolete? With GPUs of recent history, obsolescence is not a concern in a 3-year time frame. On AWS, people are still using K80s (released in 2014!). The GTX 1080 Ti, which was released 2.5 years ago, is selling for substantially above MSRP. This may change if competition in the GPU space increases and NVIDIA loses its monopoly. AWS will continually upgrade their hardware and you keep paying the same True, but this concern is mitigated by slow rate of GPU obsolescence. Hardware can be capitalized, which means you can push it to the balance sheet (for tax or valuation purposes) Can you say a little more about this? Not sure what you're getting at. Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns. This needs to be evaluated on a case-by-case basis. A poorly-capitalized start-up with an unpredictable compute workload is not a good candidate for buying on-prem. A well-capitalized start-up that consistently uses GPUs is a better candidate. Returns, breakage, warranty in case of a hardware failure Any reasonable hardware provider will include an option for 3-year warranty.
- colde 8y ago
  > Any reasonable hardware provider will include an option for 3-year warranty. Did you include the cost for that in your scenario?
  - nightfly 8y ago
    For a reasonable vendor, a 3 year warranty is have a negligible cost. (~5-10% of the price of the system)
    - ocdtrekkie 8y ago
      In fact, for most enterprise grade hardware I've ever purchased, the three year warranty tends to be pretty much built into the cost. Usually where I pay extra is to add the fourth and fifth years.
sokoloff 8y ago
> Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns. I think they used the 3-year reserved instance pricing, meaning you can’t just turn it off after a year and stop the cash flow out (at least not at that price).
- termie 8y ago
  It is possible to sell (or buy) reserved instances in a secondary market: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ri-market-general.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ri-marke...

by mbesto 8y ago

microcolonel 8y ago

> - Time (and cost) it takes to install software, drivers, etc I'd imagine this is part of having a system administrator.

vidarh 8y ago

> - Cost associated with finding an admin who understands how this thing works Managing AWS resources is not "free" either, or the market for people to handle devops work on AWS related to ongoing operations would be non-existent. My experience is when I've done consulting, clients on AWS used to end up paying more per instance than on-prem clients ended up paying per physical server. > - Time (and cost) it takes to install software, drivers, etc This is largely an initial cost of a day or two of setup when you start setting up an on-prem setup + for the first server of a totally new model. If you're buying one, then sure, you need to factor in a bit of time. If you're buying more, then if you use a competent admin the second one should be a matter of inserting boot media or (preferrably) configuring PXE booting, and picking an IP.

fucking_tragedy 8y ago
> My experience is when I've done consulting, clients on AWS used to end up paying more per instance than on-prem clients ended up paying per physical server. This has been my experience, as well. Clients who switch to AWS for cost-saving reasons will be disappointed.
- mikemotherwell 8y ago
  ... unless the use case is weird, e.g. you need massive amounts of processing power infrequently.
  - vidarh 8y ago
    There certainly are people with use-cases like that, but it is rare to find people who both have large, sudden spikes that are not better handled in other ways (e.g. combination of CDNs and micro-caching), or where the load can not be trivially smoothed out enough (e.g. there's enough work with relaxed latency requirements or that can be scheduled to off peak times, or where the spikes are not small enough relative to base-load that it still pays off to put most load on dedicated equipment and use AWS or similar to handle spikes only. People often think they have workloads like that. I've done work on plenty of setups over the years were people were certain they had lots of big spikes, but most of them only had moderate day/night cycles (and usually a lot less spiky than they thought, because they'd never bothered to measure). Only in a few cases I've seen usage patterns that justified spinning instances up/down very often, and for those cases, ironically the availability of clouds have made dedicated hosting more cost effective, because if you host everything in VMs or containers anyway, you can now get to far higher utilization levels on your on-prem/colocated setups, and be prepared to spin up some cloud instances (automatically, if you wish, it's not that much effort), and tie them into your cluster when your load warrants it. It's not suitable for all workloads, but it works surprisingly well if anything. I do see that argument a lot for using cloud providers, but the vast majority of cloud setups I've seen end up having very static workloads growing at rates that gives plenty of time to prepare.
    - 8y ago

sabalaba 8y ago

(Author here) I agree with a lot of your points, especially the fact that TCO is practically impossible to calculate, is highly subjective, and should ideally include opportunity costs which, almost by definition vary from person to person. I've definitely thought through some of your points. First I want to say that if your load is spiky you have no business buying hardware. No matter how you slice it, you're simply not going to be able to get hundreds of GPUs of throughput with an 8 GPU machine so it's not possible to get the same "product". So for those doing short bursts of hyperparameter search, stick with cloud. But, like you said, for those with high utilization and known utilization patterns, it makes a lot of sense to go on-prem. Let's just say pretty much anybody who is buying a reserved instance from AWS should consider buying hardware instead. >- Cost associated with finding an admin who understands how this thing works Pricing based on what co location facilities will provide you without much search and is pretty generous. $10k per year per server is silly high and should cover that search cost. > - Try before you buy You can try essentially this exact machine on AWS. That instance and hardware are almost exactly the same. Now of course this doesn't apply to many instance types. > - Time it takes for the box to be built, shipped and sent to the data center 5 business days is our mean lead time for that unit. Most workstations are 2 days. > - Time (and cost) it takes to install software, drivers, etc It comes with that stuff pre-installed as mentioned in the article. See Lambda Stack for driver / framework / CUDA woes: https://lambdalabs.com/lambda-stack-deep-learning-software https://lambdalabs.com/lambda-stack-deep-learning-software > - 3-year? What is the useful life of this? When does it seemingly become obsolete? I'll admit it's a long time but 3 years is about right for GPUs from my experience. I personally purchased new GPUs for my workstations in 2012 (Fermi), 2015 (Maxwell), 2017 (Pascal). > - AWS will continually upgrade their hardware and you keep paying the same Yes, but only if you never commit to a reserved instance, in which case costs are 3x higher. If you buy a reserved instance you don't get upgraded hardware. > - Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns. Yea, most of these are people who are already using GPUs really often for internal training infra and are finding it expensive. > - Returns, breakage, warranty in case of a hardware failure Our hardware comes with a 3 year parts warranty. Of course, this doesn't pay for your time lost but it's pretty rare to see parts fail and the $10k / year more than covers in-colo swap outs. I agree with all of your points. I don't think that we really disagree with much here. TCO is hard to calculate :).

mippie_moe 8y ago

Hi, Lambda engineer here (I'm one of the authors). You bring up some good points, I'd like to address some of them: Admin: Time (and cost) it takes to install software, drivers, etc These tasks aren't made unnecessary by cloud. Yes, with cloud, once you've done the work, it can be forever encoded into an image or container. However, the same applies to on-prem using container solutions like Docker. Maintenance / Capitalization / Finances: 3-year? What is the useful life of this? When does it seemingly become obsolete? With GPUs of recent history, obsolescence is not a concern in a 3-year time frame. On AWS, people are still using K80s (released in 2014!). The GTX 1080 Ti, which was released 2.5 years ago, is selling for substantially above MSRP. This may change if competition in the GPU space increases and NVIDIA loses its monopoly. AWS will continually upgrade their hardware and you keep paying the same True, but this concern is mitigated by slow rate of GPU obsolescence. Hardware can be capitalized, which means you can push it to the balance sheet (for tax or valuation purposes) Can you say a little more about this? Not sure what you're getting at. Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns. This needs to be evaluated on a case-by-case basis. A poorly-capitalized start-up with an unpredictable compute workload is not a good candidate for buying on-prem. A well-capitalized start-up that consistently uses GPUs is a better candidate. Returns, breakage, warranty in case of a hardware failure Any reasonable hardware provider will include an option for 3-year warranty.

colde 8y ago
> Any reasonable hardware provider will include an option for 3-year warranty. Did you include the cost for that in your scenario?
- nightfly 8y ago
  For a reasonable vendor, a 3 year warranty is have a negligible cost. (~5-10% of the price of the system)
  - ocdtrekkie 8y ago
    In fact, for most enterprise grade hardware I've ever purchased, the three year warranty tends to be pretty much built into the cost. Usually where I pay extra is to add the fourth and fifth years.

sokoloff 8y ago

> Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns. I think they used the 3-year reserved instance pricing, meaning you can’t just turn it off after a year and stop the cash flow out (at least not at that price).

termie 8y ago
It is possible to sell (or buy) reserved instances in a secondary market: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ri-market-general.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ri-marke...