3 ms·

I would actually say just invest in the hardware and count the asset depreciation on taxes. Further, “scaling” horizontally is rather easy if you properly separ

I would actually say just invest in the hardware and count the asset depreciation on taxes. Further, “scaling” horizontally is rather easy if you properly separate functions into different servers. For example, a few really light machines running nginx (with fastcgi cache enabled, because yes) behind an haproxy machine, your PHP/Python/JS/Ruby machines behind your nginx machines, and your DB cluster with TiDB or something behind that. You’ve removed the overhead of the container systems and the overhead of the virtualization platform. You’re no longer sharing CPU time with anyone. You’re not experiencing as many context switches or interrupts. The cost is all upfront though. You will still pay for bandwidth and power, but over time your cost should be lower. The main issue in any scenario involving real hardware is that you need staff who are competent in both hardware and Linux/UNIX systems. Many claim to be on their resumes and then cannot perform once on the job (in my experience anyway). In my opinion, one of the major reasons for the explosion of the cloud world was precisely the difficulty in building and the financial cost of building such teams. Additionally, there is a somewhat natural (and necessary) friction between application developers and systems folks. The systems folks should always pushing back and arguing for more security, more process, and fewer deployments. The dev team should always be arguing for more flexibility, more releases, and less process. Good management should then strike the middle path between the two. Unfortunately, incompetent managers have often just decided to get rid of systems people and move things into AWS land. Finally, I would just note that cloud architecture is bad for the planet as it requires over-provisioning by cloud providers, and it requires more computer power overall due to the many layers of abstraction. While anyone project is responsible for little of this waste, the entire global cloud as an aggregate is very wasteful. This bothers me and obviously likely factors as an emotional bias in my views (so large amounts of salt for all of the above).

kkielhofner 4y ago
Generally agree but in the US at least from a financial and accounting standpoint you can actually do better than the favored op-ex of cloud services with a Section 179[0] lease[1]. Caveat here, of course, is that you have to have income. The argument could be made you can develop a means to to rent physical servers pre-income, then, when it makes sense, you can either use standard depreciation -or- Section 179 on outright purchases and/or Section 179 leases. As an example, you can deploy an incredibly capable group of let's say four absolutely ridiculous completely over-provisioned $100k physical 1U machines in different colo facilities for redundancy. There are all kinds of tricks here for load balancing and failover with XYZ cloud service, DNS, anycast, whatever you want. You can go with various colo facilities that operate datacenters around the world, ship the hardware from the vendor to them, then provision them with Ansible or whatever you're into without ever seeing the facility or touching hardware. So now you have redundant physical hardware that will absolutely run circles around most cloud providers (especially for I/O), fixed costs like all you can eat bandwidth (that doesn't have the 800% markup of cloud services, etc) - no more waiting for the inevitable $50k cloud bill or trying to track down (in a panic) what caused you to exceed your configured cloud budget in a day instead of a month. Oh btw, you're not locking yourself into the goofy proprietary APIs to provision and even utilize services other than virtual machines offered by $BIGCLOUD. If you're doing any ML you can train on your own hardware or (or the occasional cloud) and run inference 24/7 with things like the NVIDIA A10. Continuous cloud rental for GPU instances is unbelievably expensive and the ROI on purchasing the hardware is typically in the range of a few months (or way ahead almost immediately with Section 179). As an example, I recently did a benchmark with the Nvidia A10 for a model we're serving and it can do over 700 inference requests/s in FP32 with under 10ms latency. With a single A10 per chassis across four healthy instances that's 2800 req/s (and could probably be tuned further). Then, if you get REALLY big, you can start getting cabinets and beyond. In terms of hardware failures as mentioned, all I can say is dual PS RAID-ed out, etc hardware is (in my experience) extremely reliable. Frankly having had multiple full cabinets of hardware in the past hardware failures were few and far between and hardware vendors will include incredibles SLAs for replacement. You notify them of the failure, they send a tech in < eight hours directly to the colo facility and replace the disk, PS, etc with the flashing light. My experience is one (good) FTE resource can easily manage this up to multiple cabinet scale. To your point, the current issue is many of these people have been snatched up by the big cloud providers and replaced (in the market) with resources that can navigate the borderline ridiculousness that is using dozens (if not more) products/services from $BIGCLOUD. I've also found this configuration is actually MUCH more reliable than most $BIGCLOUD. No more wondering what's going on with a $BIGCLOUD outage that they won't even acknowledge (and that you have absolutely no control over). Coming from a background in telecom and healthcare it's completely wild to me how uptime has actually gotten much worse with cloud providers. Usually you can just tell customers "oh the internet is having problems today" because they'll probably be seeing headlines about it but for many applications that's just totally unacceptable - and we should expect better. [0] - https://www.section179.org/section_179_deduction/ https://www.section179.org/section_179_deduction/ [1] = https://www.section179.org/section_179_leases/ https://www.section179.org/section_179_leases/
- BirAdam 4y ago
  Do you have a preferred geo-aware DNS, anycast service, or similar?

kkielhofner 4y ago

Generally agree but in the US at least from a financial and accounting standpoint you can actually do better than the favored op-ex of cloud services with a Section 179[0] lease[1]. Caveat here, of course, is that you have to have income. The argument could be made you can develop a means to to rent physical servers pre-income, then, when it makes sense, you can either use standard depreciation -or- Section 179 on outright purchases and/or Section 179 leases. As an example, you can deploy an incredibly capable group of let's say four absolutely ridiculous completely over-provisioned $100k physical 1U machines in different colo facilities for redundancy. There are all kinds of tricks here for load balancing and failover with XYZ cloud service, DNS, anycast, whatever you want. You can go with various colo facilities that operate datacenters around the world, ship the hardware from the vendor to them, then provision them with Ansible or whatever you're into without ever seeing the facility or touching hardware. So now you have redundant physical hardware that will absolutely run circles around most cloud providers (especially for I/O), fixed costs like all you can eat bandwidth (that doesn't have the 800% markup of cloud services, etc) - no more waiting for the inevitable $50k cloud bill or trying to track down (in a panic) what caused you to exceed your configured cloud budget in a day instead of a month. Oh btw, you're not locking yourself into the goofy proprietary APIs to provision and even utilize services other than virtual machines offered by $BIGCLOUD. If you're doing any ML you can train on your own hardware or (or the occasional cloud) and run inference 24/7 with things like the NVIDIA A10. Continuous cloud rental for GPU instances is unbelievably expensive and the ROI on purchasing the hardware is typically in the range of a few months (or way ahead almost immediately with Section 179). As an example, I recently did a benchmark with the Nvidia A10 for a model we're serving and it can do over 700 inference requests/s in FP32 with under 10ms latency. With a single A10 per chassis across four healthy instances that's 2800 req/s (and could probably be tuned further). Then, if you get REALLY big, you can start getting cabinets and beyond. In terms of hardware failures as mentioned, all I can say is dual PS RAID-ed out, etc hardware is (in my experience) extremely reliable. Frankly having had multiple full cabinets of hardware in the past hardware failures were few and far between and hardware vendors will include incredibles SLAs for replacement. You notify them of the failure, they send a tech in < eight hours directly to the colo facility and replace the disk, PS, etc with the flashing light. My experience is one (good) FTE resource can easily manage this up to multiple cabinet scale. To your point, the current issue is many of these people have been snatched up by the big cloud providers and replaced (in the market) with resources that can navigate the borderline ridiculousness that is using dozens (if not more) products/services from $BIGCLOUD. I've also found this configuration is actually MUCH more reliable than most $BIGCLOUD. No more wondering what's going on with a $BIGCLOUD outage that they won't even acknowledge (and that you have absolutely no control over). Coming from a background in telecom and healthcare it's completely wild to me how uptime has actually gotten much worse with cloud providers. Usually you can just tell customers "oh the internet is having problems today" because they'll probably be seeing headlines about it but for many applications that's just totally unacceptable - and we should expect better. [0] - https://www.section179.org/section_179_deduction/ https://www.section179.org/section_179_deduction/ [1] = https://www.section179.org/section_179_leases/ https://www.section179.org/section_179_leases/

BirAdam 4y ago
Do you have a preferred geo-aware DNS, anycast service, or similar?