3 ms·

Maybe so, I haven't made this move before, which is part of the reason I'm asking here, as unlikely as it may be that "someone who is in this position would be

Maybe so, I haven't made this move before, which is part of the reason I'm asking here, as unlikely as it may be that "someone who is in this position would be asking on HN." Sorry for being vague on all of this. Regardless: 1.) No specific discount, this is standard, public pricing on major commercial clouds, with some tricks applied like reserved instances, which is part of the reason I am not worried about low prices first and then ballooning pricing later, 2.) Our operations and maintenance and sustaining are in fact more expensive that anyone would reasonably expect, I believe partially due to vendors getting their hooks in decades ago and expanding scope and increasing prices, I've walked the data center floors and coded with the ops teams. It is #2 that is really hitting us. I have personally moved several apps as a PoC and you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs and autoscaling. I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally. Thank you for the response.

late2part 8y ago
You really should contact me in confidence. We’ve done almost exactly this and the data I have is compelling.
mmt 8y ago
> I believe partially due to vendors getting their hooks in decades ago and expanding scope and increasing prices I would be surprised if this isn't the vast majority of it, with the rest being the over-provisioning you mention later. Mostly, it's tough to gauge, because each effect is multiplicative. I'll go so far as to say that, somewhere between 10 and 15 years ago, that any hardware support you've been paying to vendors can be considered as complete waste [1]. Worse, it's often a percentage (i.e. multiplier) of original purchase that goes one indefinitely, even though the value of that gear falls rapidly. Also, (for that 90% you believe you can move to the cloud), if you have any proprietary hardware, you're paying an eye-watering multiple of its commodity equivalent (even considering additional labor, if any). The above is easiest to observe in storage. There's not all that much mystery around how much the components cost and how reliable they are, so cost estimation for an equivalent commodity solution is easy enough. Today, even commodity/free no longer means "no frills" in terms of software features, but I expect that's less relevant when comparing to cloud providers, anyway. > you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs I very much would. Just remember you can smaller VMs yourself, cheaper than cloud providers, bringing your staff up to speed on more modern skills, so long as you don't fall into a vendor's trap. VMWare's model where the software ends up cost is the same order of magnitude as the hardware is such a trap, but so is software that's free but requires expensive professional services to set up and maintain (OpenStack, historically, though I'm not sure if that's still true). > and autoscaling. This, actually, is the rare case where cloud providers can save really save money, if your duty cycle (for lack of a better term) is under 10% of so. Other comments in the thread mentioned this, as well. Unfortunately, if you never autoscale to (approximately) zero, then you're grossly over-paying for that base load, which can wipe out those savings. The "hybrid cloud" model addresses this, as well as avoiding all-or-nothing thinking. Ultimately, if you're optimizing (not necessarily exclusively, of course) for cost and want it to last, you'll have to change the whole organization's way of thinking. Avoiding vendor lock-in is key, be it proprietary hardware, software, or cloud providers. Also, encourage good engineering practice/culture, without getting overly focused on the latest fashionable tool. (A trite example being saving on labor through automation but buggy automation causing an eye-watering AWS bill). [1] There's some non-zero value to any actual replacement hardware, but that's negligible if considering what you actually got, net of any markup for being propreitary.
exikyut 8y ago
I'd been wondering about the possibility of #2 when I made my other original reply a few hours ago. Thanks for clarifying. The first thing I'm reminded of is https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-.... It's from 2009 but this image jumped out at me: https://www.backblaze.com/blog/wp-content/uploads/2009/08/cost-of-a-petabyte-chart.jpg https://www.backblaze.com/blog/wp-content/uploads/2009/08/co... Let me guess... you're using kit from the vendor at the bottom of that list? :) No need to Y/N; I'll assume Y because the sentiment's probably representatively/loosely correct. The commentator your/this reply is to points out an interesting aspect: > What likely would not be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect. This sort of jumped out at me a bit, it's a very reasonable assumption to make, particularly as you say you're an old-school operation, which suggests to me that the tech components are heavily politicized due to their management by non-technical types. Well... the landscape is indeed changing, as clarified by the 9-year old article above. So major shifting-around of vendor selection is going to ruffle (and maybe squash) a few feathers, but these "old world" tech enterprises cannot reasonably expect everything to keep going indefinitely because commodity-hardware based solutions have become so accessible that it's harder than ever to sugarcoat proprietary systems to even mildly-technical management, particularly with compute and storage, so the writing is on the wall, and you can use that in your favor. (Well, except clouds. That seems to be where the old-world's shifted to. Heh.) And the thing is, for a $B operation (for really any value of $B..$BBB), the cost-benefit absolutely goes in your favor if you build everything out and run it yourself. Plus, if you really are a $BBB operation, you can build your own servers and network kit. As in, you get to start answering questions like what features you want on the motherboards (no Ethernet since you're supplying your own 10G kit? sure thing... oh you don't want any onboard SATA either? okay), how you want the systems to boot (build your own secure boot implementation!), what BIOS/EFI features you want (you want the half-hour RAM test disableable for your staging boxes? fine), how long you'd like your enclosures to be (36" is really long but your racks can handle the length you'll be able to fit a remarkable number disks in 4U), what firmware optimization suggestions you have for your HDDs (a writeable vendor area in SMART config for how many seconds you want each disk to pause for before spinning up so systems come up slowly and don't kill their breakers? done)... and all of these things will directly translate to cost savings. If there's one thing that differentiates "old-world" datacenters from shots of FAANG, Backblaze, etc, it's that all the big players' servers look really, realy boring, particularly the enclosures. (Well, okay, the blue LEDs make up for it. :P) And this is because there's almost nothing in those machines that isn't needed. Can you get away with removing VGA? (You almost certainly can with virtualization hosts, even if you're just doing KVM or Xen on top of Linux.) Would doing short 24V or 48V runs help? (Complex, but could lower power consumption.) All of these things serve to save money. How much exactly probably depends on what sorts of overheads you're currently dealing with; just switching to barebones systems that do have a few bells and whistles but are still very minimalist may get you an appreciable saving that all the obscure things I've listed here may not be necessary (ie may have negative/negligible/not-worth-it cost-benefit analyses). I guess my question is at this point, is the decision scoped down to "what vendor are we selecting" or is building your own DC still on the table? I so hope it is, because IMO that's where the cost savings really is. Hm, maybe you could go "I have one more idea I want to try before we commit to this, it'll require $$$,$$$" and then go and replace your most expensive storage array or something :P that could be pretty convincing... > I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally. I think it's cool you're really getting behind this and making your job description part of your identity :P hopefully this is respected and you don't get too heavily stepped on. There are some really interesting comments in this thread, and a fair bit of reiteration that clouds are generally expensive solutions. I hold this sentiment myself, FWIW, despite admittedly having a LOT less experience with the practical side of things than everyone else here. This was what I was getting at with the "the offer will be what the buying team wants to hear" - at this point it's almost like the vendor sales are actors in a real-time movie that's exactly what the buying team wants to watch. On top of that one vendor is really going to be as good as another (I've been mentioning GCE and AWS, I forgot Azure.) Cannot keep writing as I must go, may continue later

late2part 8y ago

You really should contact me in confidence. We’ve done almost exactly this and the data I have is compelling.

mmt 8y ago

> I believe partially due to vendors getting their hooks in decades ago and expanding scope and increasing prices I would be surprised if this isn't the vast majority of it, with the rest being the over-provisioning you mention later. Mostly, it's tough to gauge, because each effect is multiplicative. I'll go so far as to say that, somewhere between 10 and 15 years ago, that any hardware support you've been paying to vendors can be considered as complete waste [1]. Worse, it's often a percentage (i.e. multiplier) of original purchase that goes one indefinitely, even though the value of that gear falls rapidly. Also, (for that 90% you believe you can move to the cloud), if you have any proprietary hardware, you're paying an eye-watering multiple of its commodity equivalent (even considering additional labor, if any). The above is easiest to observe in storage. There's not all that much mystery around how much the components cost and how reliable they are, so cost estimation for an equivalent commodity solution is easy enough. Today, even commodity/free no longer means "no frills" in terms of software features, but I expect that's less relevant when comparing to cloud providers, anyway. > you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs I very much would. Just remember you can smaller VMs yourself, cheaper than cloud providers, bringing your staff up to speed on more modern skills, so long as you don't fall into a vendor's trap. VMWare's model where the software ends up cost is the same order of magnitude as the hardware is such a trap, but so is software that's free but requires expensive professional services to set up and maintain (OpenStack, historically, though I'm not sure if that's still true). > and autoscaling. This, actually, is the rare case where cloud providers can save really save money, if your duty cycle (for lack of a better term) is under 10% of so. Other comments in the thread mentioned this, as well. Unfortunately, if you never autoscale to (approximately) zero, then you're grossly over-paying for that base load, which can wipe out those savings. The "hybrid cloud" model addresses this, as well as avoiding all-or-nothing thinking. Ultimately, if you're optimizing (not necessarily exclusively, of course) for cost and want it to last, you'll have to change the whole organization's way of thinking. Avoiding vendor lock-in is key, be it proprietary hardware, software, or cloud providers. Also, encourage good engineering practice/culture, without getting overly focused on the latest fashionable tool. (A trite example being saving on labor through automation but buggy automation causing an eye-watering AWS bill). [1] There's some non-zero value to any actual replacement hardware, but that's negligible if considering what you actually got, net of any markup for being propreitary.

exikyut 8y ago

I'd been wondering about the possibility of #2 when I made my other original reply a few hours ago. Thanks for clarifying. The first thing I'm reminded of is https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-.... It's from 2009 but this image jumped out at me: https://www.backblaze.com/blog/wp-content/uploads/2009/08/cost-of-a-petabyte-chart.jpg https://www.backblaze.com/blog/wp-content/uploads/2009/08/co... Let me guess... you're using kit from the vendor at the bottom of that list? :) No need to Y/N; I'll assume Y because the sentiment's probably representatively/loosely correct. The commentator your/this reply is to points out an interesting aspect: > What likely would not be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect. This sort of jumped out at me a bit, it's a very reasonable assumption to make, particularly as you say you're an old-school operation, which suggests to me that the tech components are heavily politicized due to their management by non-technical types. Well... the landscape is indeed changing, as clarified by the 9-year old article above. So major shifting-around of vendor selection is going to ruffle (and maybe squash) a few feathers, but these "old world" tech enterprises cannot reasonably expect everything to keep going indefinitely because commodity-hardware based solutions have become so accessible that it's harder than ever to sugarcoat proprietary systems to even mildly-technical management, particularly with compute and storage, so the writing is on the wall, and you can use that in your favor. (Well, except clouds. That seems to be where the old-world's shifted to. Heh.) And the thing is, for a $B operation (for really any value of $B..$BBB), the cost-benefit absolutely goes in your favor if you build everything out and run it yourself. Plus, if you really are a $BBB operation, you can build your own servers and network kit. As in, you get to start answering questions like what features you want on the motherboards (no Ethernet since you're supplying your own 10G kit? sure thing... oh you don't want any onboard SATA either? okay), how you want the systems to boot (build your own secure boot implementation!), what BIOS/EFI features you want (you want the half-hour RAM test disableable for your staging boxes? fine), how long you'd like your enclosures to be (36" is really long but your racks can handle the length you'll be able to fit a remarkable number disks in 4U), what firmware optimization suggestions you have for your HDDs (a writeable vendor area in SMART config for how many seconds you want each disk to pause for before spinning up so systems come up slowly and don't kill their breakers? done)... and all of these things will directly translate to cost savings. If there's one thing that differentiates "old-world" datacenters from shots of FAANG, Backblaze, etc, it's that all the big players' servers look really, realy boring, particularly the enclosures. (Well, okay, the blue LEDs make up for it. :P) And this is because there's almost nothing in those machines that isn't needed. Can you get away with removing VGA? (You almost certainly can with virtualization hosts, even if you're just doing KVM or Xen on top of Linux.) Would doing short 24V or 48V runs help? (Complex, but could lower power consumption.) All of these things serve to save money. How much exactly probably depends on what sorts of overheads you're currently dealing with; just switching to barebones systems that do have a few bells and whistles but are still very minimalist may get you an appreciable saving that all the obscure things I've listed here may not be necessary (ie may have negative/negligible/not-worth-it cost-benefit analyses). I guess my question is at this point, is the decision scoped down to "what vendor are we selecting" or is building your own DC still on the table? I so hope it is, because IMO that's where the cost savings really is. Hm, maybe you could go "I have one more idea I want to try before we commit to this, it'll require $$$,$$$" and then go and replace your most expensive storage array or something :P that could be pretty convincing... > I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally. I think it's cool you're really getting behind this and making your job description part of your identity :P hopefully this is respected and you don't get too heavily stepped on. There are some really interesting comments in this thread, and a fair bit of reiteration that clouds are generally expensive solutions. I hold this sentiment myself, FWIW, despite admittedly having a LOT less experience with the practical side of things than everyone else here. This was what I was getting at with the "the offer will be what the buying team wants to hear" - at this point it's almost like the vendor sales are actors in a real-time movie that's exactly what the buying team wants to watch. On top of that one vendor is really going to be as good as another (I've been mentioning GCE and AWS, I forgot Azure.) Cannot keep writing as I must go, may continue later